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I. 


INTRODUCTION 


As  Phillips  (1997)  stated,  “When  examining  current  publications  related  to 
ROI  . . .  most  models  and  representations  of  the  ROI  process  ignore,  or  provide  very  little 
insight  into  the  two  key  elements  essential  to  developing  the  ROI:  Isolating  the  effects  of 
training,  and  converting  data  to  monetary  values”  (p.  xiv).  This  project  attempted  to  do 
this  for  the  Navy’s  Training  with  Industry  (TWI)  program  so  that  the  Naval  Supply 
Systems  Command  (NAVSTJP)  can  make  better  informed  decisions  regarding  investing 
resources,  including  manpower,  into  the  program. 

A.  BACKGROUND 

The  United  States  Navy  Supply  Corps  can  trace  its  inception  to  1795  when  the 
first  Supply  Corps  officers  were  assigned  to  support  the  original  six  frigates  of  the  U.S. 
Navy.  At  this  time,  the  original  duties  and  responsibilities  of  the  Navy  Supply  Corps 
included  logistics  support.  The  Supply  Corps’  overall  mission  today,  “delivering 
sustained  global  logistics  capabilities  to  the  Navy  and  Joint  Warfighter,”  has  not  changed 
significantly  (Naval  Supply  Systems  Command  [NAVSUP],  n.d.-a).  However,  the 
Supply  Corps  has  grown  to  over  3,500  active  duty  and  reserve  component  officers. 
Supply  Corps  officers  are  serving  not  just  on  ships,  but  on  submarines,  in  the 
expeditionary  environment,  and  in  joint  operations,  both  in  the  United  States  and  abroad. 
Today’s  Supply  Corps  officers  are  trained  to  be  well-rounded  naval  officers,  and  they 
receive  specific  training  in  one  of  the  three  supply  lines  of  operations:  supply  chain 
management,  acquisition  management,  and  operational  logistics  (NAVSUP,  n.d.-a). 

A  successful  career  in  the  Supply  Corps  requires  skills  and  talents  to  support 
maritime,  contingency,  and  expeditionary  environments.  Through  job  rotation,  an  officer 
picks  up  technical  and  leadership  skills  which  help  to  gain  “Big  Navy”  insight  to  logistics 
operations  while  providing  exposure  to  managing  logistics  from  the  unit  level  to  the 
strategic  level.  With  each  billet  assigned,  Supply  Corps  officers  build  professional 
development  and  gain  experience  to  better  support  the  fleet.  Supply  Corps  officers  learn 
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problem  solving  techniques  through  both  training  and  education  (Office  of  Supply  Corps 
Personnel,  2011). 

From  the  ranks  of  ensign  through  lieutenant,  the  Supply  Corps  career  path 
develops  tactical  junior  officers  who  learn  how  the  Navy  is  organized  and  how  it  operates 
at  the  unit  level.  Lieutenant  commanders  are  often  sent  to  graduate  school,  which,  along 
with  prior  experience,  helps  build  and  refine  leadership  and  logistics  skills  to  help  them 
take  on  key  leadership  and  operational  billets  as  commanders  (Office  of  Supply  Corps 
Personnel,  2011). 

The  Supply  Corps  often  awards  what  is  termed  “top-tier  assignments”  to 
individuals  who  sustain  superior  performance,  accept  challenging  billets,  maintain  a  solid 
reputation,  and  network  with  individuals  in  the  supply  community.  One  example  of  a  top- 
tier  assignment  is  selection  for  a  TWI  internship  (Office  of  Supply  Corps 
Personnel,  2011). 

Every  year,  four  accomplished  lieutenant  (0-3)  and  lieutenant  commander  (0-4) 
Supply  Corps  officers  are  selected  for  participation  in  the  TWI  program.  To  be  eligible 
for  selection,  the  officer  is  (a)  an  0-3  or  0-4,  (b)  has  a  master’s  degree  or  will  obtain  one 
in  the  calendar  year  following  the  board,  (c)  completed  two  operational  tours,  and  (d)  has 
a  projected  rotation  date  (PRD)  in  the  calendar  year  following  the  selection  board 
(NAVSUP,  2013).  The  TWI  program  provides  the  opportunity  for  those  selected  to  be 
part  of  the  operations  of  well-regarded  companies  with  large  logistic  footprints,  including 
Starbucks,  FedEx,  The  Home  Depot,  and  ExxonMobil.  The  TWI  program  is  designed  “to 
provide  training  and/or  development  of  skills  in  private  sector  procedures  and  practices 
not  available  through  existing  military  or  advanced  civilian  education  programs  or  other 
established  training  and  education  programs”  (Department  of  Defense  [DOD],  2007). 
During  this  one-year  assignment,  the  Supply  Corps  officer  executes  a  permanent  change 
of  station  (PCS)  to  the  company  headquarters  to  work  in  the  field  with  real-world 
business  issues;  assists  with  quality  and  safety  audits,  and  contingency  and  strategic 
planning;  and  develops  a  solid  understanding  of  how  the  company  manages  its  supply 
chain  (Adams,  2012).  After  one  year  in  the  TWI  program,  the  officer  has  a  three-year 
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payback  tour  in  a  select  billet  with  the  intent  to  bring  best  practices  from  the  corporate 
world  to  the  fleet. 


1.  Department  of  Defense  Training  with  Industry  Policy 

DOD  Instruction  1322.06  establishes  the  policy  for  the  TWI  program.  This 
program,  along  with  other  fellowships  and  scholarships,  is  to  “fulfill  a  present  need, 
anticipated  requirement,  or  future  capability  that  contributes  to  the  effectiveness  of  the 
respective  Military  Department  and  the  Department  of  Defense”  (DOD,  2007).  While 
participating  in  the  program,  the  DOD  pays  the  participant  normal  pay  and  allowances. 
The  payback  requirement  for  the  participant  is  to  complete  at  least  three  times  the  length 
of  the  TWI  tour.  Therefore,  a  one-year  TWI  tour  requires  a  three-year  payback 
(DOD,  2007). 

DOD  Instruction  1322.06  goes  on  to  specify  that  military  personnel  may  be 
accepted  into  TWI  only  under  the  following  circumstances: 

•  In  recognition  of  outstanding  performance  in  their  fields 

•  To  undertake  a  project  that  may  be  of  value  to  the  United  States 

•  For  development  of  their  recognized  potential  for  future  career  service 

•  To  acquire  a  skill,  knowledge,  or  ability  to  fulfill  a  present  need, 
anticipated  requirement,  or  future  capability  that  contributes  to  the 
effectiveness  of  the  respective  Military  Department  and  contributes  to  the 
transformation  of  the  Department  of  Defense.  (DOD,  2007,  p.  8) 


The  DOD  Instruction  1322.06  also  states  the  criteria  for  an  assignment  to  the  TWI 
program  must  include  the  following: 

•  There  must  be  an  existing  Military  Component  need  or  desired  future 
capability  fulfilled  by  virtue  of  the  experience  gained. 

•  There  must  be  either  a  follow-on  utilization  tour  or  assignment  to  which 
the  individual  shall  be  assigned,  or  a  clear,  documented  future  need  for  the 
skill  that  is  gained. 

•  If  individuals  completing  the  TWI  are  not  immediately  placed  in  a 
utilization  tour  or  assignment,  the  Secretary  concerned  shall  keep 
administrative  oversight  of  the  individuals  and  their  gained  skills  for 
utilization  at  a  time  determined  by  the  Military  Component. 
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•  The  TWI  tour  or  assignment  should  not  exceed  12  months  in  length.  Any 
TWI  tour  or  assignment  that  will  exceed  12  months  must  be  approved  by 
the  Secretary  concerned. 

•  The  proposed  TWI  tour  or  assignment  must  meet  professional 
development  requirements. 

•  There  must  be  a  written  agreement  between  the  private  sector  host,  the 
employee,  and  the  DOD  Component  concerned  before  the  start  of  the  TWI 
assignment. 

•  The  education  or  training  to  be  received  or  the  research  to  be  performed 
by  a  fellowship,  scholarship,  TWI,  or  grant  recipient  must  be  designed  to 
qualify  the  recipient  to  satisfy  a  requirement  or  potential  requirement  of 
the  Department  of  Defense,  contribute  to  the  recipient’s  recognized 
potential  for  career  service,  or  constitute  a  contribution  to  a  project  of 
value  to  the  United  States,  (pp.  9-10) 

Each  branch  of  the  military,  with  the  exception  of  the  United  States  Marine  Corps 
(USMC),  participates  in  the  TWI  program.  These  programs  are  described  in  the  following 
section. 


2.  United  States  Army  Training  with  Industry 

The  U.S.  Army  has  sent  both  commissioned  and  non-commissioned  officers 
through  the  TWI  program  since  the  1970s  (United  States  Army  Human  Resources 
Command  [HRC],  2016).  The  Army’s  objective  in  participating  in  the  TWI  program  is  to 
develop  officers  and  non-commissioned  officers  (NCOs)  who  understand  industry  and 
who  gain  new  management  practices  through  the  private  sector.  Partnering  with  industry 
helps  the  Army  to  “support  marketing,  public  affairs,  artificial  intelligence,  physical 
security,  and  finance”  (HRC,  2016).  Over  40  companies  partner  with  the  Army  and 
accept  TWI  participants,  including  the  National  Football  League,  Caterpillar,  and  Google 
(Enlisted  Personnel  Management  Directorate  [EPMD],  2017c). 

Officers  and  NCOs  who  want  to  participate  in  the  TWI  program  submit 
applications  to  Military  Schools  Branch,  where  a  panel  selects  top  candidates  and  submits 
the  names  to  a  leader  development  division  for  final  approval.  Prospective  participants 
have  demonstrated  the  potential  for  a  long-term  career  in  the  Army  and  must  have  been 
recently  promoted,  or  qualified  for  promotion  (EPMD,  2017b).  The  Army  has  a  total  of 

50  TWI  allocations  per  year.  Each  assignment  is  6  to  12  months  (EPMD,  2017a). 
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The  Army  requires  its  participants  to  follow  a  training  plan  with  objectives, 
perform  self-study  actions,  and  complete  specific  activities.  Both  mid-term  and  final 
reports  are  due  to  the  Army  training  coordinator  (HRC,  2016).  The  payback  for  Army 
officers  and  NCOs  is  a  three-year  active  duty  assignment  and  a  two-year  utilization  tour. 

3.  United  States  Air  Force  Education  with  Industry 

The  Air  Force’s  program  is  called  Education  with  Industry  (EWI).  The  EWI 
program  is  open  to  officers  and  civilians  of  the  rank  0-3  to  0-4  or  GS  11  to  GS  13  (Air 
Force  Institute  of  Technology  [AFIT],  2015).  The  Air  Force  started  this  program  in  1947 
because  the  Air  Force  recognized  a  need  for  its  personnel  to  understand  the  workings  of 
the  defense  industry  and  for  Air  Force  personnel  and  industry  to  communicate  service 
needs  to  each  other.  Companies  that  partner  with  the  Air  Force  for  EWI  include  The 
Boeing  Company,  Microsoft  Corporation,  and  Honeywell  (AFIT,  2015). 

Officers  are  nominated  for  the  EWI  program  by  their  senior  rater  (Secretary  of  the 
Air  Force,  2009).  The  nomination  is  reviewed  by  a  team  who  then  forward  the 
applications  for  the  recommended  individuals  to  a  selection  board.  Civilians  submit  their 
own  application  for  consideration  of  acceptance  into  the  EWI  program.  Civilians  can 
apply  through  Career  Field  Teams,  using  an  Air  Force  Civilian  Competitive  Development 
Nomination  Form.  Unlike  officers,  who  typically  PCS  to  an  EWI  location,  civilians 
partner  with  companies  within  close  proximity  to  their  home  location  (Secretary  of  the 
Air  Force,  2009).  The  number  of  persons  selected  for  the  program  varies  from  year  to 
year,  but  it  is  usually  between  30  and  40  personnel  (K.  Hansen,  personal  communication, 
February  16,  2017).  Assignments  for  the  EWI  program  are  10  months  in  length  and  incur 
a  three-year  commitment  after  completion  (United  States  Air  Force,  2009;  Secretary  of 
the  Air  Force,  2009). 

The  Department  of  the  Air  Force  (2009)  specifies  the  requirements  for  the  Air 
Force  participants  to  include  developing  a  work  plan  to  identify  the  objectives  and  to 
guide  activities  to  complete  during  the  program.  The  work  plan  is  a  flexible  document  to 
enable  participants  to  capitalize  on  unforeseen  opportunities  that  arise  and  are  deemed 
beneficial.  During  the  program,  participants  are  required  to  submit  three  individual 
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reports  containing  information  on  activities  performed,  benefits  derived,  and  personal 
observations.  Participants  also  complete  a  research  paper  during  the  EWI  program 
(Department  of  the  Air  Force,  2009). 

4.  United  States  Navy  and  Marine  Corps  Secretary  of  the  Navy  Tours 
with  Industry 

Aside  from  the  four  participants  sent  through  the  TWI  program  each  year  from 
the  Navy  Supply  Corps,  in  October  of  2015,  the  Navy  initiated  Secretary  of  the  Navy 
Tours  with  Industry  (SNTWI)  (Department  of  the  Navy  [DON],  2015).  The  secretary  of 
the  Navy  lays  out  a  policy  for  both  officers  (0-3  or  above)  and  enlisted  (E-6  or  above)  to 
work  11  to  12  months  with  industry  to  familiarize  themselves  with  business  planning, 
organization,  management  techniques,  innovations,  and  best  practices  (DON,  2015).  At 
SNTWI’ s  inception,  Amazon  and  FedEx  partnered  with  the  Navy  (Chief  of  Naval 
Personnel  Public  Affairs,  2015). 

Participation  in  the  program  is  voluntary;  however,  personnel  are  first  nominated 
and  then  selected  from  their  naval  community.  As  of  2015,  the  Navy  had  five  participants 
in  the  SNTWI  program  (Chief  of  Naval  Personnel  Public  Affairs,  2015).  The  Navy 
intends  to  grow  the  program  to  15-20  officers  and  15-20  enlisted  each  year. 

Participants  submit  periodic  reports  documenting  experiences  and  observations. 
Additional  briefing  to  Naval  leaders  may  be  required  (DON,  2015).  The  Marine  Corps, 
under  the  Department  of  the  Navy  (DON),  is  required  to  establish  an  SNTWI  program. 
No  information  has  been  found  on  the  Marine  program. 

5.  Training  Oversight 

The  military  components  are  authorized  to  send  service  members  to  training  in 
nongovernmental  facilities,  like  TWI,  EWI,  and  SNTWI,  under  Section  2013  of  Title  10 
of  the  U.S.  Code.  However,  a  Government  Accountability  Office  (GAO)  report,  GAO- 
12-367,  called  out  the  Army,  Navy,  and  Air  Force  for  lack  of  oversight  of  their  TWI 
programs.  The  report  specifically  lists  four  main  reasons: 

•  Not  all  of  the  services  conduct  periodic  program  review. 
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•  The  services  do  not  have  clear  guidance  as  to  what  qualifies  as  a  follow-on 
utilization  tour. 

•  The  services  do  not  know  their  overall  program  costs,  including  both 
direct  and  indirect  costs,  and  therefore  it  is  difficult  to  know  whether  these 
program  are  cost-effective. 

•  Some  of  the  services  do  not  have  written  agreements  or  memoranda  of 
understanding  with  the  non-DOD  host  organizations  providing  the 
educational  opportunity  that  spell  out  both  parties’  roles  and 
responsibilities.  (Farrell,  2012,  pp.  23-24) 


The  Naval  Supply  Systems  Command  (NAVSUP)  invests  in  the  TWI  partnerships 
with  the  Navy  Supply  Corps.  However,  there  is  no  tool  to  evaluate  the  return  on 
investment  (ROI)  from  the  TWI  program.  This  MBA  project  evaluated  the  Supply  Corps 
TWI  program  from  an  ROI  perspective.  Through  analysis  of  the  partnerships  and  a 
determination  of  ROI,  NAVSUP  will  be  able  to  make  better  informed  decisions  regarding 
investing  in  the  TWI  program. 

B.  OBJECTIVES 

Several  methodologies  for  ROI  were  analyzed  to  determine  the  best  method  to  use 
for  the  Navy  to  determine  its  ROI. 

The  primary  question  is 

•  What  are  the  costs  and  benefits  of  the  TWI  program  to  the  Navy? 
The  secondary  questions  are 

•  What  is  the  retention  rate  for  TWI  fellows  after  the  three-year  utilization 
tour? 

•  Does  the  utilization  tour  fill  the  correct  billet  and  meet  the  professional 
requirement? 

•  Do  the  skills/competencies  obtained  through  TWI  meet  the  DOD’s 
guidance? 

•  What  are  the  effects  of  the  TWI  program  on  the  officer’s  promotion  and 
career? 
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Through  the  course  of  this  research,  the  best  methodology  to  use  to  measure  the 
ROI  of  the  Navy’s  TWI  program  and  an  example  for  the  calculation  is  provided. 

C.  SCOPE,  LIMITATIONS,  AND  ASSUMPTIONS 

The  scope  of  the  project  is  limited  to  Navy  Supply  Corps  officers  who  have 
completed  a  TWI  internship.  The  program  has  been  around  since  2006,  and  the  number 
of  participants  is  24.  A  portion  of  the  data  was  collected  via  survey  of  the  Supply  Corps 
officers  who  have  completed  the  TWI  program.  Because  no  military  component 
effectively  determines  the  cost  and  benefit  of  TWI  programs,  research  into  best  practices 
is  limited  to  using  private  sector  cases. 
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II.  LITERATURE  REVIEW 


A.  WHAT  IS  ROI  AND  WHY  USE  ROI? 

Return  on  investment  (ROI)  is  extensively  used  by  organizations  to  make  better 
business  decisions.  Determining  the  ROI  can  be  a  challenging  process  but  is  overall 
beneficial  for  the  organization. 

1.  Background/History 

In  the  1980s,  the  development  of  total  quality  management  (TQM)  was,  according 
to  Chmielewski  and  Phillips  (2002),  a  management-focused  “shift  toward  a  performance- 
oriented  approach  to  business  practices”  (p.  225).  Because  of  TQM,  business 
organizations  experienced  growth  and  increased  profits  while  fostering  the  production  of 
top  quality  products.  In  1993,  the  National  Partnership  for  Reinventing  Government 
(NRP)  emulated  this  concept  to  create  an  efficient  and  low  cost  operating  government. 
Chmielewski  and  Phillips  go  on  to  explain  that  Congress  passed  the  Government 
Performance  and  Result  Act  of  1993  (GPRA),  “which  required  government  agencies  to 
submit  strategic  plans  and  performance  measurements  to  justify  their  budgets”  (p.  226). 
According  to  the  authors,  GPRA  required  agencies  to  seek  out  ways  to  measure 
performance  so  they  could  be  more  accountable,  and  led  to  the  popular  usage  of  the  ROI 
methodology. 

2.  Benefits  of  ROI 

There  are  many  reasons  why  ROI  should  be  used  according  to  Phillips  (2007). 
First,  the  increase  in  training  budget  requires  accountability  and  closer  observation  for 
investment  justification.  Second,  top  executives  care  about  the  costs  and  benefits.  This 
goes  back  to  justification  for  investment  in  competing  programs.  Third,  ROI  is  a  familiar 
term.  Most  managers  with  a  business  degree  have  learned  and  used  it  in  an  academic 
environment,  so  they  will  demand  it  on  the  job.  Additionally,  there  are  many  benefits  to 
measuring  the  ROI.  According  to  Phillips  (2007),  these  are  the  top  five  benefits: 

•  Measure  contribution.  ROI  methodology  is  most  commonly  used  to 
demonstrate  benefits  of  training  to  business  because  of  its  accuracy  and 
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reliability.  It  shows  the  benefits  versus  the  costs  of  the  program  in  terms  of 
dollar  values.  It  answers  the  question:  Is  the  return  worth  the  investment? 

•  Establish  priorities.  Calculating  the  ROI  will  ensure  visibility  on  positive 
or  negative  returns,  which  will  enable  executive  leaders  to  prioritize 
resources  in  fiscally  constrained  environments.  Having  insight  on  ROI  will 
provide  decisions  makers  with  relevant  facts  to  make  thorough  and 
accurate  conclusions  as  to  keep  or  eliminate  inefficient  programs  or 
programs  with  lesser  impact. 

•  Focus  on  results.  Because  ROI  “requires  instructional  designers, 
facilitators,  participants,  and  support  groups  to  concentrate  on  measurable 
objectives”  (p.7),  the  overall  effectiveness  of  the  training  program  will 
improve. 

•  Earn  respect  of  senior  executives  and  sponsors.  Senior  executives  will  be 
able  to  make  better  decisions  because  they  will  be  able  to  see  the 
connection  of  certain  training  programs  to  business  goals  in  actual 
monetary  value. 

B.  ROI  IMPLEMENTATION  CHALLENGES 

Phillips  (2007)  acknowledges  that  some  organizations  see  ROI  as  the  answer  to 
accountability  and  training  justification,  while  others  see  it  as  an  inaccurate  and 
burdensome  method.  This  is  especially  evident  when  ROI  is  used  inappropriately.  An 
inappropriate  application  of  ROI  is  when  applying  ROI  does  not  meet  a  balance  of 
“feasibility,  simplicity,  credibility  and  soundness”  (Chmielewski  &  Phillips,  2002, 
p.  228).  Although  applying  the  ROI  methodology  provides  benefits,  it  remains  a 
challenge  for  many  organizations  to  use.  Some  of  these  challenges  highlighted  by  Phillips 
(2007)  are  also  relevant  to  the  Navy. 

•  Cost  and  time.  To  effectively  conduct  ROI  requires  time  and  money,  a 
commodity  already  scarce  in  most  organizations.  Generally,  it  adds  3-5% 
to  the  program  budget,  but  may  be  offset  in  the  end  by  the  benefit  gained 
through  the  program. 

•  Lack  of  skills  and  orientation  for  human  resources  department  staff.  ROI 
is  not  a  widely  taught  subject  and  is  normally  not  part  of  training  for  a  job. 
A  typical  program  is  focused  on  the  learning  process  and  not  on  measuring 
results. 

•  Faulty  needs  assessment.  Many  training  programs  are  in  place  due  to  the 
want  of  a  manager  at  that  specific  time  and  may  not  be  thought  through 
with  the  need  of  the  training  program.  Minimal  need  leads  to  minimal 
benefit. 
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•  Fear.  The  fear  of  failure  and  the  unknown  will  keep  learning  and 
development  (L&D)  professionals  from  utilizing  the  ROI  process. 
Additionally,  the  ROI  process  usually  initiates  the  traditional  fear  of 
change. 

•  Discipline  and  planning.  To  successfully  implement  ROI  requires 
significant  discipline  and  planning  from  the  L&D  team.  If  there  is  no 
direct  pressure  from  the  boss  for  results,  the  L&D  team  may  not  dedicate 
the  time  needed  for  an  effective  evaluation. 

•  False  assumptions.  L&D  teams  are  prevented  from  using  the  ROI  process 
because  of  assumptions,  such  as  that  the  “impact  of  learning  cannot  be 
accurately  measured”  (p.  7),  executive  leaders  do  not  require  ROI,  and 
learning  is  complex  and  cannot  be  accurately  measured. 

Schmidt  (2017)  points  out  that  although  many  training  program  evaluations 
measure  the  initial  reaction  of  the  program  participants  for  positive  feedback,  few 
conduct  actual  ROI  of  the  overall  program.  This  is  due  to  a  misconception  of  the 
complexity  of  the  method,  the  additional  cost  and  time  required  to  measure  the  ROI 
(Schmidt,  2017).  Many  business  professional  find  it  difficult  to  produce  hard  monetary 
returns,  or  to  demonstrate  program  value.  Most  training  produces  positive  evidence  for 
providing  the  training,  but  does  not  produce  quantitative  financial  justification  or  a 
compelling  ROI  metric  to  support  the  training  budget  request.  Although  possible  to 
measure  objectively,  quantitatively  and  credibly  putting  effective  measurements  in  place 
takes  careful  planning,  time,  and  effort  (Schmidt,  2017). 

While  the  commercial  world  and  government  sector  share  similar  challenges 
relating  to  ROI  data,  the  government  also  has  unique  challenges.  Most  organizations 
within  the  government  do  not  invest  in  ROI  training  for  the  workforce.  Some 
organizations  within  the  government  do  not  have  a  human  resources  department,  creating 
a  challenge  for  organizations  to  effectively  use  ROI  methodology  to  measure  programs. 
Chmielewski  and  Phillips  (2002)  said,  “For  an  ROI  process  to  be  useful,  it  must  balance 
many  issues  such  as  feasibility,  simplicity,  credibility,  and  soundness”  (p.  228). 
Additionally,  in  the  government,  in  order  for  the  ROI  to  be  useful,  three  targeted 
audiences  must  accept  the  ROI  process.  The  audiences  are  human  resource  practitioners, 
managers,  and  researchers.  Chmielewski  and  Phillips  (2002)  further  explain,  “the 
challenge  is  to  develop  acceptable  requirements  for  an  ROI  process  within  the 
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government  system  that  will  satisfy  researchers  and,  at  the  same  time,  please  practitioners 
and  executive  managers”  (p.  228).  Contrary  to  many  beliefs,  this  is  possible. 

C.  ROI  METHODOLOGY  USED  BY  DIFFERENT  SERVICES 

Although,  research  showed  no  record  of  ROI  for  the  TWI  program  in  any  of  the 
military  services,  the  DOD  conducted  numerous  ROI  measurements  for  similar  programs. 
These  include  the  following: 

•  An  Analysis  of  the  Return  on  Investment  of  Navy  Enterprise  Resource 
Planning  as  Implemented  Navy-Wide  FY04-FY15.  A  Naval  Postgraduate 
School  (NPS)  MBA  Professional  Report  written  by  Robert  Kovack  Jr.  and 
Philip  R.  Lindley.  http : //calhoun . nps . edu/bitstream/handle/ 10945/10758/ 
llJun%255FKovack%255FMBA.pdf?sequence=l&isAllowed=y 

Used  classic  ROI  formula  to  compare  cost-benefit  from  Navy  to 
private  sector. 

•  Evaluating  Navy’s  Funded  Graduate  Education  Program:  A  Return-on- 
Investment  Framework.  Prepared  by  the  National  Defense  Research 
Institute  (RAND)  and  written  by  Kristy  N.  Kamarck,  Harry  J.  Thie, 
Marisa  Adelson,  and  Heather  Krull.  http://www.rand.org/content/dam/ 
rand/pubs/monographs/2010/RAND  MG995.pdf 

Used  the  ROI  framework  to  measure  the  ROI  for  funded  graduate 
education  in  the  Navy’s  surface  warfare  officer  and  the 
meteorology  and  oceanography  communities. 

•  Calculating  Return  on  Investment  for  U.S.  Department  of  Defense 
Modeling  and  Simulation.  Prepared  by  the  Defense  Acquisition  University 
(DAU)  and  written  by  Ivar  Oswalt,  Tim  Cooley,  William  Waite,  Elliot 
Waite,  Steve  “Flash”  Gordon,  Richard  Severinghaus,  Jerry  Feinberg,  and 
Gary  Lightner.  http://www.dau.mil/pubscats/pubscats/AR%20Journal 
/arj58/Oswalt%2QARJ  58.pdf 

Developed  and  used  measuring  metrics  to  produce  an  ROI-like 
(qualitative  and  monetary  value)  result  that  the  DOD  can  use  to 
prioritize  modeling  and  simulation  investments. 

•  An  Analysis  of  Return  on  Investment  Options  for  the  USMC  Distance 
Learning  Program.  A  Naval  Postgraduate  School  (NPS)  thesis  written  by 
Jamie  E.  Clark,  http://calhoun.nps.edu/bitstream/handle/10945/9353/ 
OOMar  Clark.pdf?sequence=l&isAllowed=y 

Used  the  cost  benefit  analysis  process  and  the  ROI  model  to 
calculate  the  cost  and  benefit  of  the  USMC  Distance  Learning 
program. 
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D.  ROI  METHODOLOGY 


This  project  reviewed  three  different  ROI  methodologies  for  training  programs. 
Kirkpatrick’s  four  levels  model  has  served  as  a  foundation  for  many  training  and 
evaluation  measurements  in  the  commercial  sector,  including  the  other  two  models 
examined  namely  Schmidt’s  seven  steps  model  and  the  Phillips’  five-levels  ROI 
framework.  These  three  models  are  discussed  in  this  section. 

1.  Kirkpatrick’s  Four  Levels  Model 

As  defined  by  Kirkpatrick  and  Kirkpatrick  (2013),  program  evaluation  is 
important  to  determine  continuity,  improvement,  existence  justification,  compliance, 
effectiveness,  and  alignment  to  business  strategy.  It  is  important  to  have  support  from 
managers  and  leaders  at  the  executive  levels  to  ensure  success  of  the  evaluation. 

When  using  the  model,  Kirkpatrick  recommends  reversing  the  model  (levels  4,  3, 
2,  and  1)  during  the  planning  and  executing  the  steps  in  a  chronological  order  during  the 
monitoring.  The  model  is  shown  in  Tables  1  and  2,  with  further  descriptions  from 
Kirkpatrick  and  Kirkpatrick  (2013). 


Table  1.  Planning  Phase  of  the  Kirkpatrick  Model.  Source:  Kirkpatrick  & 

Kirkpatrick  (2013,  p.  19). 


Level  4:  Results 

Find  out  business  needs  and  opportunities 

Level  3:  Behavior 

Define  non-negotiable  required  behaviors 

Level  2:  Learning 

Identify  required  knowledge,  skills  and  attitude  to  perform  the 
required  behaviors 

Level  1:  Reaction 

Determine  learning  environment  that  will  facilitate  obtaining 
required  knowledge  and  skills 
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Table  2.  The  Kirkpatrick  Model.  Source:  Kirkpatrick  &  Kirkpatrick 

(2013,  p.  6). 


Level  1:  Reaction 

To  what  degree  participants  react  favorably  to  the  learning  event 

Level  2:  Learning 

To  what  degree  participants  acquire  the  intended  knowledge,  skills  and  attitudes  based  on  their  participant  in 
the  learning  event 

Level  3:  Behavior 

To  what  degree  participants  apply  what  they  learned  during  training  when  they  are  back  on  the  job 

Level  4:  Results 

To  what  degree  targeted  outcomes  occur,  as  a  result  of  learning  event(s)  and  subsequent  reinforcement 

Last,  after  following  the  steps  outlined  in  Tables  1  and  2,  the  Return  on 
Expectation  (ROE)  should  be  measured.  This  is  the  degree  of  satisfaction  the  training 
initiative  delivers  to  the  stakeholders.  Was  the  expectation  identified  in  the  planning 
phase  met? 

2.  Schmidt’s  Seven  Steps  Model 

Schmidt  (2017)  wrote  an  article  with  a  seven-step  approach  to  measuring  the 
value  of  professional  training  and  ROI.  Measuring  the  value  help  business  professional 
justify  the  training  when  organizations  are  moving  toward  “zero-based  budgeting”  (p.l) 
environment,  merging  and  acquisitioning,  changing  financial  leaders,  and  reassessing 
working  capital.  The  following  are  the  seven  steps  structured  approach  (Schmidt,  2017). 

a.  Step  1:  Recruit  and  Use  a  Training  Advisory  Team 

Measuring  the  ROI  and  improvement  to  the  ROI  of  a  training  program  should  not 
fall  solely  on  the  training  manager  (Schmidt,  2017).  Training  ROI  is  a  team  effort  and 
works  best  when  there  is  a  training  advisory  team  that  meets  consistently  to  develop, 
evaluate,  and  measure  the  training  requirement  of  the  company. 

b.  Step  2:  Link  Training  ROI  to  Existing  Performance  Measurement 
Systems 

Schmidt  (2017)  recommends  reviewing  performance  measurement  already  in 
place  at  the  organization  such  as  balanced  score  cards,  financial  metrics,  customer 
metrics,  or  internal  business  process  metrics.  If  current  metrics  are  implemented  poorly, 
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improve  upon  the  current  metrics.  If  current  metrics  are  implemented  exceptionally  well, 
use  the  existing  metrics.  The  training  ROI  metrics  should  be  linked  to  these  metrics. 
Speaking  the  same  language  as  the  organization  will  ensure  faster  buy  in  and 
measurement  relevancy. 

c.  Step  3:  Measure  Level  1  of  the  Kirkpatrick  Model  ( Response  to  the 
Training ) 

Aim  for  a  return  to  the  organization  and  the  participant.  Use  Kirkpatrick’s  level 
one  of  training  measurement  by  asking  the  participant  specific  questions  to  determine  on- 
the-job  impact  and  benefits,  personal  and  professional  value,  and  needed  course 
improvement  (Schmidt,  2017). 

d.  Step  4:  Measure  Level  2  of  the  Kirkpatrick  Model  (Learning  and 
Retention ) 

To  attribute  specific  learning  to  the  training,  the  pre-course  and  post-course 
testing  must  be  in  place.  Schmidt  (2017)  expounds  on  Kirkpatrick  by  having  participants 
complete  a  second  post-course  test  to  measure  retention  of  the  training. 

e.  Step  5:  Measure  Level  3  and  4  of  the  Kirkpatrick  Model  ( Changed 
Behavior  and  Business  Impact) 

Training  is  valuable  to  the  organization  “only  when  it  leads  to  changes  in 
behavior  that  help  meet  business  objectives”  (Schmidt,  2017,  p.8).  The  training  advisory 
team  and  various  managers  are  an  asset  for  measuring  skill  and  performance 
improvements  like  (a)  “percentage  of  participants  applying”  the  new  skill,  (b)  “tangible 
improvements  in  productivity  or  efficiency,”  (c)  “tangible  improvements  to  quality  of 
work,”  (d)  “new  abilities  or  skills  following  training,”  and  (e)  “links  between  changed 
behaviors  and  business  objectives”  (p.8). 

/.  Step  6:  Measure  All  the  Training  Costs 

Calculate  the  “cost  per  successful  participant”  (Schmidt,  2017,  p.8)  in  addition  to 
the  total  cost  of  the  training.  This  will  “show  management  how  training  ROI  should 
improve  in  the  future”  (p.8).  Distinguish  between  the  fixed  training  cost  and  the  variable 
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training  cost  to  show  true  benefit  of  specific  training.  Also  consider  opportunity  costs  and 
other  hidden  costs  in  this  calculation. 

g.  Step  7:  Deliver  the  Full  Training  Report  Card 

Use  the  clearest  and  most  direct  approach,  which  is  the  classic  ROI  formula: 

Training  ROI  =  (Total  returns  -  Training  cost)  /  Training  cost 

or 

Training  ROI  =  (Total  returns  /Training  cost)  -  1 

Also  include  all  metrics  used  because  different  managers  may  want  to  use 
different  metrics  (Schmidt,  2017). 

3.  Phillips’  Five-Level  ROI  Framework 

Phillips’  five-level  ROI  framework  combines  the  four  features  from  the 
Kirkpatrick  model  with  a  way  to  measure  the  investment  and  benefit,  the  return  on 
investment  (see  Figure  1).  Phillips’  measurement  focus  ensures  a  systematic  evaluation  of 
the  transfer  of  learning  from  the  training  program  and  its  impact  to  the  organization 
(Chmielewski  &  Phillips,  2002). 


Level  of  Evaluation 

Measurement  Focus 

1.  Reaction  &  Planned  Action 

Measures  employee  satisfaction  with  the  program  and 
captures  planned  action. 

II.  Learning 

Measures  changes  in  employee  knowledge,  skills,  and  attitudes 
related  to  the  program. 

III.  Job  Applications 

.Measures  changes  in  on-the-job  behavior/job  processes. 

IV.  Business  Results 

Measures  changes  ‘in  business-impact  variables. 

V.  Return  on  Investment  (ROI) 

Compares  program  benefits  to  the  costs. 

Figure  1.  Five  Levels  of  ROI  Evaluation.  Source:  Chmielewski 

&  Phillips  (2002). 
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Chmielewski  and  Phillips  (2002)  suggest  four  factors  to  consider  before  planning 
the  ROI  evaluation:  “evaluation  purposes,  instruments,  levels,  and  timing”  (p.  228).  The 
evaluation  purpose  will  “determine  the  scope  of  the  evaluation”  (p.  228),  which  leads  to 
the  types  of  measurement  instruments  used  and  choosing  which  data  to  capture.  It  is  also 
important  to  determine  what  levels  are  appropriate  for  data  collection  and  the  timing  of 
the  data  collection.  These  are  key  factors  for  an  effective  data  collection  plan.  The  major 
steps  for  Phillips’  framework  are  “data  collection,  isolating  program  effects,  converting 
data  to  monetary  values,  tabulating  program  cost,  calculating  ROI,  and  identifying 
intangible  benefits”  (p.  229). 

a.  Data  Collection 

Collect  hard  data  (e.g.,  input,  output,  time,  and  cost)  and  soft  data  (e.g.,  command 
climate,  attitudes,  and  performance  habits)  using  the  best  data  collection  method(s)  for 
the  setting,  specific  program,  time  allotted,  and  budget  constraints  (see  Figure  2) 
(Chmielewski  &  Phillips,  2002). 
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Method 

Level 

Application 

Surveys 

3 

•Determines  the  degree  to  which  employees  have  utilized  various 
aspects  of  the  program. 

•Responses  are  often  developed  on  a  sliding  scale  and  usually 
represent  attitudinal  data 

Questionnaires 

3,  4 

•Uncovers  specific  applications  of  HR  initiatives/programs. 

•Employees  provide  responses  to  a  variety  of  types  of  open-ended 
and  forced-response  questions 

On-the-job  1 

observation 

3 

•Captures  actual  skill  application  and  use. 

•Observations  are  particularly  useful  in  customer  service  initiatives  and  are 
more  effective  when  the  observer  is  either  invisible  or  transparent. 

Post-program  1 

interviews 

3 

•Determines  the  extent  of  program  effectiveness. 

•Allows  for  probing  to  uncover  specific  applications. 

Focus  groups 

3 

•Determines  the  degree  to  which  a  group  of  employees  are  involved  in  the 
program  and  the  extent  to  which  the  program  effects  day-to-day  operations. 

Program  1 

Assignments 

3,  4 

•Employees  complete  the  assignment  on  the  job,  utilizing  components  or 
initiatives  of  the  program. 

Action  plans 

3,  4 

•Developed  in  programs  and  are  implemented  on  the  job. 

•A  follow-up  provides  evidence  of  program  success. 

Performance  1 

Contracts 

3,  4 

•Developed  where  the  employees,  managers,  and  executives  all  agree  on 
specific  outcomes  of  the  program. 

Follow-up  Sessions  3,  4 

•Captures  evaluation  data  as  well  as  presents  additional  learning  material. 

Performance  1 

Monitoring 

4 

•  Useful  when  various  performance  records  and  operational  data  are  1 

examined  for  improvement.  1 

Figure  2.  Data  Collection  Methods  and  Application  to  the  ROI  Evaluation. 

Source:  Chmielewski  &  Phillips  (2002). 
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b.  Isolating  the  Effects  of  the  Training  Program 

An  essential  step  that  is  often  overlooked  is  the  isolating  of  the  effects  of  the 
program.  Comparison  between  participants  going  through  the  program  versus  individuals 
that  did  not  demonstrates  the  direct  attribution  the  training  program  has  on  the  outcome. 
This  is  to  ensure  that  the  ROI  calculation  for  the  training  program  is  accurate.  Figure  3 
shows  strategies  used  by  organizations  to  isolate  the  effects  of  programs. 


Strategy  Type  of  Effect  Isolated 

Control  group  *One  group  participates  in  the  program  while  a  similar  group  does  not. 

•The  difference  in  performance  of  the  two  groups  is  attributed  to  the  program. 

Trend  lines  *Used  to  project  the  value  of  specific  output  variables,  if  the  program  has  not  been 

undertaken. 

•The  projection  is  compared  to  the  actual  data  after  the  program  and  the  difference 
represents  the  estimate  of  the  impact. 

Forecasting  model  •Used  when  mathematical  felationships  between  input  and  output  variables  are  known. 

•With  this  approach,  the  output  variable  is  predicted  with  the  assumption  that  the 
program  is  not  conducted. 

•The  actual  performance  of  the  variable  after  the  program  is  then  compared  with  the 
forecast  to  estimate  program  impact. ' 

Employees  estimate  'Employees  are  provided  with  the  total  amount’ of  improvement  oh  a  pre-  and 
post-program  basis. 

•Asked  to  indicate  the  percent  of  improvement  that  is  actually  related  to  the  program! 

Managers  estimate  “Managers  are  presented  with  the  total  amount  of  Improvement  and  are  asked  to 
indicate  the  percentage  related  to  the  program  Senior  management  estimate 

•  Managers  provide  an  estimate  or  'adjustment'  to  reflect  the  portion  of  the 
improvement  related  to  the  program. 

Experts  estimate  ’Because  the  estimates  are  based  on  previous  experience,  the  experts  must  be  familiar 
with  the  type  of  program  implementations  and  the  specific  situation. 

Subordinates 

estimate  •Subordinates  of  the  employees  affected  by  the  program  assess  if  other  variables 

changed  in  the  work  climate  could  have  influenced  output  performance. 

Extraneous  factors  ‘Other  influencing  factors  are  identified  and  their  impact  estimated  or  calculated! 

The  unexplained  portion  of  the  output  is  then  attributed  to  the  HR  program  . 

Customer  input  ‘Describe  the  extent  to  which  the  components  of  the  program  have  influenced  their 
decision  to  use  a  product  or  service. 


Figure  3.  Strategies  to  Isolate  the  Effects  of  the  Training  Program.  Source: 

Chmielewski  &  Phillips  (2002). 

c.  Converting  Data  to  Monetary  Values 

It  is  important  to  convert  data  collected  in  level  4  (Figure  1)  of  the  framework  to 

monetary  values  in  order  to  calculate  the  ROI.  Chmielewski  and  Phillips  (2002) 

recommend  ten  strategies  that  may  be  used  to  convert  data  collected  into  monetary  value. 
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The  strategy  selected  to  convert  data  to  monetary  value  “depends  on  the  type  of  data 
collected  and  the  situation”  (p.  229).  For  example,  if  the  data  collected  is  an  output  data, 
this  data  can  be  “converted  to  profit  contribution  or  cost  saving”  (p.  232)  to  the 
organization.  Examples  of  data  and  types  of  conversion  are  in  Figure  4. 


Unrt  of  Data 

Type  of  Conversion 

Output  data 

•Data  is  converted  to  profit  contribution  or  cost  savings. 

•Output  increases  are  converted  to  monetary  value  based  on  their  contribution  to  profit 
or  cost  reduction. 

Quality  data 

•The  cost  of  quality  is  calculated  and  quality  improvements  are  directly  converted  to 
cost  savings. 

Time  data 

•For  programs  where  employee  time  is  saved,  wages  and  benefits  are  used  for  the 
value  of  time. 

•Since  many  programs  focus  on  improving  the  time  required  to  complete  projects, 
processes,  or  daily  activities,  the  value  of  time  is  important  to  consider. 

Organizational  1 

cost  data 

•  Historical  costs  and  current  records  are  used  when  available  for  a  specific  variable. 
•Organizational  cost  data  are  utilized  to  establish  the  specific  value  of  an  improvement. 

Estimate  of  value 

•When  available,  internal  and  external  experts  may  be  used  to  estimate  a  value  for 
an  improvement. 

•The  credibility  of  the  estimate  hinges  on  the  expertise  and  reputation  of  the  individual. 

Estimate  of  cost 

•External  databases  are  sometimes  available  to  estimate  the  value  or  cost  of 
data  items. 

•Government,  industry,  and  research  databases  can  provide  important  information  for 
these  values. 

Employee  estimates  1 

of  improvement 

•Employees  estimate  the  value  of  the  data  item.  1 

•They  must  be  able  to  provide  a  value  for  the  improvement.  1 

Manager  estimates  1 

of  improvement 

•  Provide  estimates  when  they  are  both  willing  and  capable  of  assigning  value  to  the 
improvement. 

•  Useful  when  employees  are  not  fully  capable  of  providing  this  input  or  in  situations 
where  supervisors  need  to  confirm  or  adjust  employee’s  estimate. 

Senior  management  1 

estimate  of  1 

improvement 

•Provide  estimates  on  the  value  of  an  improvement  when  they  are  willing  to  offer  1 

estimates.  1 

Expert  estimates  1 

of  improvement 

•  Estimates  made  by  HR  professionals  involved  in  the  program  are  used  to  determine 
the  value  of  output  data. 

•Estimates  must  be  provided  on  an  unbiased  basis. 

Figure  4.  Strategies  to  Convert  Data  to  Monetary  Value  in  an  ROI  Evaluation. 

Source:  Chmielewski  &  Phillips  (2002). 
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d.  Tabulating  Program  Cost 

Consolidate  all  related  costs  of  the  program  in  order  to  calculate  the  ROI.  Some  of 
the  costs  to  consider  are  (a)  design  and  development  of  the  program,  (b)  organization  of 
the  program,  (c)  implementation  of  the  program,  (d)  salaries  plus  employee  benefits,  (e) 
training  employee  on  using  the  program,  (f)  contractors  who  help  facilitate  the  program, 
(g)  evaluating  the  program,  and  (h)  administrative  and  overhead  (Phillips,  1997). 

e.  Calculating  the  ROI 

Chmielewski  and  Phillips  (2002)  used  the  program  benefits  and  cost  to  calculate 
ROI.  The  formula  they  use  for  the  benefit/cost  ratio  (BCR)  is  as  follows  (p.  231): 

BCR  =  Program  Benefit/Program  Costs 
The  formula  for  the  net  program  benefit  is  as  follows: 

Net  Program  Benefit  =  Program  Benefit  -  Program  Cost 
The  formula  for  the  ROI  is  as  follows: 

ROI  (%)  =  Net  Program  Benefit/Program  Cost  x  100s 

The  ROI  in  some  programs  may  be  higher  than  100%.  For  example,  an  ROI  of 
650%  means  that  for  each  dollar  invested  in  the  training  program,  the  organization 
received  $6.50  “dollars  in  return  after  the  cost  of  the  program  had  been  recovered” 
(Phillips,  1997,  p.  154).  In  personnel  systems,  the  ROI  may  be  lower.  This  is  due  to 
intangible  benefits  that  may  not  be  possible  to  capture  in  monetary  value  (Phillips,  1997). 

/.  Identifying  Intangible  Benefits 

Chmielewski  and  Phillips  (2002)  said  all  attempts  should  be  used  to  convert  “hard 
and  soft  data  to  monetary  value”  (p.  232),  but  if  the  “conversion  is  too  subjective  or 
inaccurate,  and  the  resulting  conversions  lose  credibility  in  the  process”,  then  the  data 
should  be  “listed  as  an  intangible  benefit”  (p.  233).  An  example  provided  by  Phillips 
(1997)  is  a  selling  skills  program.  In  this  program,  the  attempt  was  made  to  convert 
customer  satisfaction  data  into  monetary,  but  the  “value  lost  credibility”  so  “customer 
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satisfaction  was  reported  as  intangible  benefit”  (p.  172).  Some  common  intangible 
benefits  are  “increased  job  satisfaction,  increased  organizational  commitment,  improved 
teamwork,  improved  customer  service,  reduced  complaints,  and  reduced  conflicts” 
(Chmielewski  &  Phillips,  2002,  p.  232).  Intangible  benefits  may  be  the  most  important 
benefit  for  some  programs,  especially  leadership  skills,  so  they  must  be  included. 

4.  Other  Considerations  for  ROI  Measurement 

In  addition  to  using  one  of  the  models  discussed  previously,  Bennington  and 
Laffoley  (2012)  offers  five  steps  for  human  resources  (HR)  personnel  measuring  the  ROI 
to  take  into  consideration  to  ensure  a  more  effective  demonstration  of  ROI. 

a.  Step  1:  Have  That  Critical  Conversation 

Have  an  initial  discussion  with  senior  level  leaders  to  identify  what  success  means 
to  them  in  regard  to  the  training  program. 

b.  Step  2:  Make  Smiley  Sheets  More  Robust 

An  initial  reaction  sheet  can  be  biased,  so  to  minimize  this,  re-evaluate 
participants’  reactions  three  to  six  months  after  the  program,  including  concrete  examples 
of  how  they  applied  their  experience  to  their  work.  When  possible,  assess  the  reaction  of 
their  supervisors,  peers,  and  subordinates. 

c.  Step  3:  Include  Real  Business  Challenges  in  the  Program 

Incorporate  actual  and  relevant  projects  into  the  training  program.  Implementing 
such  deliverables  can  reap  significant  ROI  for  executive  leadership  and  demonstrate  the 
importance  of  the  training  program.  This  is  a  win/win  for  both  the  participant  and  the 
organization. 

d.  Step  4:  Integrate  Learning  Programs  into  the  Organization ’s 
Performance  Management  System  and  Hold  All  Stakeholders 
Accountable 

Ensuring  that  the  “performance  appraisal  goals  reflect  those  targeted  outcomes  for 
participants  and  supervisors”  (Bennington  &  Laffoley,  2012,  p.ll)  will  “increase  the 
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probability  that  the  targeted  outcomes  of  the  programs  remain  a  focus  area  for 
participants”  (p.  11).  This  increases  their  motivation  to  learn  as  much  as  possible  from  the 
program.  Track  the  before  and  after  performances  of  participants  and  do  not  be  afraid  to 
compare  their  performance  to  that  of  equivalent  employees.  Think  creatively  in 
developing  metrics  to  measure  the  success  of  the  program. 

e.  Step  5:  Assign  Participants  Actual  Projects  after  the  Learning 
Experience 

Assign  actual  projects  to  participants  after  the  training  and  establish  check-point 
measurements  to  monitor  the  effectiveness  of  training  programs.  This  will  reap  benefit 
for  organizations  (Bennington  &  Laffoley,  2012). 

E.  EXAMPLES  OF  ROI  MEASUREMENT 

In  a  2007  article,  Phillips  showed  examples  of  published  ROI  studies.  These  are 
from  a  variety  of  applications  and  generally  displayed  high  ROI,  although  not  always  the 
case.  These  are  shown  in  Figure  5. 
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Sample  of  Published  ROI  Studies 


Case  Study  Name 

Measuring  the  ROI: 

Key  Impact  Measures: 

ROI 

Cracker  Box 

Performance  Management 
(Restaurant  Chain) 

A  variety  of  measures,  such  as  productivity, 
quality,  time,  costs,  turnover,  and  absenteeism 

298%' 

Federal  Information  Agency 

Internal  Graduate  Degree  Program 
(Federal  Agency) 

Retention,  individual  graduate  projects 

153%4 

Healthcare,  Inc. 

Sexual  Harassment  Prevention 
(Health  Care  Chain) 

Complaints,  turnover,  absenteeism,  job 
satisfaction 

1 052%2 

Imperial  National  Bank 

Executive  Leadership 

Development  (Financial) 

Team  projects,  individual  projects,  retention 

62%2 

International  Car  Rental 

First  Level  Leadership 

Development  (Auto  Rental 
Company) 

Various  measures  -  at  least  two  per  manager 

105%7 

MetroTransit 

Absenteeism  Control/Reduction 
Program  (Major  City) 

Absenteeism,  customer  satisfaction 

882%2 

Midwest  Electric 

Stress  Management  Program 
(Electric  Utility) 

Medical  costs,  turnover,  absenteeism 

320%2 

National  Crushed  Stone 

Skill-Based  Pay  (Construction 
Materials  Firm) 

Labor  costs,  turnover,  absenteeism 

805%2 

National  Steel 

Safety  Incentive  Plan  (Steel 
Company) 

Accident  frequency  rate,  accident  severity  rates 

379%2 

Nations  Hotel 

Executive  Coaching  (Hotel  Chain) 

Cost  reduction,  sales  growth,  operating 
efficiency,  retention,  and  customer  satisfaction 

221  %5 

Nextel  Communications 

Diversity  (Communications 
Company) 

Retention,  employee  satisfaction 

163%6 

Southeast  Corridor  Bank 

Retention  Improvement  (Financial 
Services) 

Turnover,  staffing  levels,  employee  satisfaction 

258%3 

United  Petroleum 

E-Leaming  (Petroleum) 

Sales 

206%2 

VA  Sunshine  Healthcare 

Network 

Competency  Development 
(Veteran's  Health  Administration) 

Time  savings,  work  quality,  faster  response 

159%4 
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Figure  5.  Sample  of  Published  ROI  Studies.  Source:  Phillips  (2007). 


An  excellent  example  of  the  usage  of  the  Kirkpatrick’s  four  levels  model  can  be 

found  at  Kirkpatrick  and  Kirkpatrick  (2006).  In  this  example,  Gap  Inc.  identified  a  need 

for  leadership  training  for  store  managers  at  Gap,  Gap  Kids,  Banana  Republic,  and 

international  divisions  of  Gap,  Inc.  The  training  program  to  fill  this  void  was  Leadership 

Training  for  Supervisors  (LTS),  which  focuses  on  “supervisory  and  leadership  skills” 
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(Kirkpatrick  &  Kirkpatrick,  2006,  p.  26)  specifically  on  how  to  “influence  and  interact 
with  store  employees”  (p.  26).  Gap  Inc.  partnered  with  Blanchard  Training  and 
Development  to  deliver  a  three-day  program  starting  with  general  managers,  area 
managers,  district  managers,  and  regional  managers.  In  1995  and  1996,  LTS  was 
available  for  store  managers.  The  program  continues  today  due  to  its  success.  The 
program’s  effectiveness  was  measured  on  all  four  levels,  as  listed  in  Table  3. 


Table  3.  Evaluation  Methods.  Adapted  from  Phillips  (2007), 
Kirkpatrick  &  Kirkpatrick  (2006). 


Level  1:  Reaction 

Qualitative  and  quantitative  data  were  collected  using  the  LTS  Program  Evaluation  form,  collected  from  each 
participant  at  the  end  of  the  program  to  capture  their  reaction 

Level  2:  Learning 

Randomly  selected  17-percent  of  the  participants  were  given  "fil l-i n-the-blank"  LTS  questionnaire  to  evaluate 
their  learning.  Before  the  training,  participants  were  told  they  may  be  selected  to  contributing  to  an 
anonymous  questionnaire. 

Level  3:  Behavior 

Randomly  selected  17-percent  of  the  participants  were  given  an  LTS  Post-Program  Survey  to  evaluate  the  short 
term  behavior  change.  A  different  LTS  Post-Program  survey  was  given  to  these  participants'  direct  supervisors. 
For  long-term  behavior  changes,  leadership  skill  assessments  were  administer  to  participants  before  the 
training  and  six  to  nine  months  after  the  training.  Results  are  compared  to  measure  percent  of  changes. 

Level  4:  Results 

Study  was  conduct  to  see  if  there  was  a  link  between  the  learned  leadership  skill  and  the  "store  sales, 
employee  turnover  rates,  and  shrinkage"  (p.49). 

After  interpretation  of  data  in  each  area,  the  training  program  was  determined  a 
success  because  there  was  a  positive  reaction  to  the  program,  new  skills  and  knowledge 
were  learned,  the  new  skills  and  knowledge  were  used  to  improve  performance,  and  the 
stores’  business  was  positively  impacted.  The  only  criticism  for  this  example  is  that  no 
ROI  calculation  was  measured,  but  the  form,  survey,  and  assessment  used  in  this  case 
study  provided  examples  to  emulate. 

F.  UNIQUE  NAVY  MANNING  SYSTEM 

Billet  funding  for  the  Navy  is  prioritized  based  on  mission  requirements,  available 
funding,  and  personnel  executability  (Chief  of  Naval  Operations  [CNO],  2016).  A 
requirement  for  shore  duty  is  based  on  a  valid,  approved  workload.  When  considering 
shore  manpower,  requirements  for  memorandums  of  understanding  (MOU)  are 
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considered,  like  the  MOUs  for  TWI  partners.  Budget  submitting  offices  (BSO)  validate 
and  identify  requirements  they  would  like  authorized.  Requirements  are  not  authorized 
until  they  are  funded  by  the  Office  of  the  Chief  of  Naval  Operations  (OPNAV).  Due  to 
fiscal  constraints,  the  Navy  may  be  restricted  in  authorizing  all  validated  billets. 
Authorization  must  never  exceed  manpower  requirements  (CNO,  2016). 

NAVSUP  is  the  community  manager  for  the  U.S.  Navy  Supply  Corps.  Like  the 
rest  of  the  Navy,  the  Supply  Corps  manages  officers  and  billets  through  designation 
codes.  These  designations  can  describe  billet  duties;  requirements  to  qualify  for  billets; 
and  skills,  qualifications,  or  traits  obtained  throughout  a  career. 

A  way  to  identify  specific  skills  or  experiences  required  or  obtained  from  billets  is 
through  the  Navy  officer  occupational  classification  system  (NOOCS).  The  NOOCS  “is 
the  method  to  identify  skills,  education,  training,  experience  and  capabilities”  (DON, 
2017a,  p.  3).  It  is  a  management  tool  for  how  communities  manage  personnel  in  terms  of 
assigning  to  training  billets,  promoting  to  the  next  rank,  distributing  certain  qualified 
personnel  amongst  the  fleet,  or  utilizing  to  help  guide  professional  development  of 
officers.  The  description  of  general  billet  duties  is  found  in  the  Navy  officer  billet 
classification  (NOBC)  code.  For  example,  a  code  of  1515  represents  inventory  control, 
which  identifies  a  billet  that  may  require  assembling,  compiling,  and  analyzing  inventory 
data.  NOBC  codes  identify  billet  requirements  to  fill  vacancies.  The  code  also  represents 
officer  experience  gained  in  a  particular  billet  (DON,  2017a). 

According  to  the  Department  of  the  Navy  (2017a),  subspecialty  codes  identify 
professional  disciplines  secondary  to  an  officer’s  primary  specialty.  A  primary  specialty, 
or  designator,  is  a  four-digit  number  identifying  officers  within  an  officer  community  and 
specialization.  For  instance,  a  Supply  Corps  officer  designated  as  a  “3100”  signifies  a 
staff  officer,  specialized  in  supply.  A  subspecialty  code,  or  secondary  discipline,  in  the 
Supply  Corps  could  be  1302,  which  identifies  Supply  Chain  Management.  Sub  specialties 
can  be  gained  through  meeting  educational  requirements  or  obtaining  core  skill 
requirements.  Requirements  may  be  quantifiable  skills,  traits,  and  experiences  that  must 
be  possessed  to  acceptably  perform  in  a  coded  billet  (DON,  2017a). 
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An  additional  qualification  designation  (AQD)  is  assigned  to  an  officer  based  on 
qualified  experience  and  education  (DON,  2017a).  AQDs  denote  other  qualifications  and 
skills  not  already  identified  in  a  community  designator,  NOBC  or  subspecialty  code.  An 
AQD  identifies  a  more  specific  qualification  required  by  a  billet  or  awarded  to  an  officer. 
An  AQD  of  SQ1  represents  a  Supply  Corps  officer  who  is  submarine  qualified  or  a  billet 
coded  with  SQ1  requires  a  Supply  Corps  officer  who  is  submarine  qualified  (DON, 
2017a). 

This  system  helps  in  the  identification  of  special  skills  required  for  a  billet.  It  also 
identifies  special  skills  of  officers.  While  the  TWI  program  does  not  award  a  subspecialty 
code,  it  does  award  participants  an  AQD  of  24G.  The  24G  AQD  is  titled  SECNAV  Tours 
with  Industry  and  is  awarded  after  an  officer  has  successfully  participated  in  the  SNTWI 
program  (DON,  2017a).  Although  the  Supply  Corps  officers  are  participating  in  TWI  and 
not  SNTWI,  the  two  programs  are  similar  enough  that  TWI  participants  are  also  awarded 
24G  (P.  Knauss,  personnel  communication,  March  6,  2017).  According  to  the  eSUPPO 
app  Frequently  Asked  Questions,  participants  may  request  a  service  school  code  of  code 
800.  Service  school  codes  identify  on  an  officer  data  card  (ODC)  service  schools 
attended.  The  800  code  identifies  the  completion  of  a  management  short  course  at  various 
schools  and  universities  around  the  country  (DON,  2017b). 

The  other  way  to  identify  skills  gained  from  the  TWI  program  on  a  military 
record  is  through  NOBC.  The  NOBC  associated  with  a  TWI  billet  is  3289.  This 
represents  a  student.  The  Manual  of  Navy  Officer  Manpower  and  Personnel 
Classifications  (DON,  2017a)  describes  the  code  as  follows:  “Participates  in  full-time 
studies  in  general  or  special  schools  (civilian  or  service)  or  performs  student  duties  under 
scholarship  of  military  commands.”  There  is  no  specific  NOBC  billet  that  TWI 
participants  go  to  for  their  utilization  tour. 

G.  CHAPTER  SUMMARY 

ROI  methodology  is  a  popular  tool  used  to  justify  training  investment  and 
maintain  accountability.  There  are  many  benefits  (i.e.,  focuses  on  results,  enables  better 
prioritization,  and  earns  respect  of  senior  executives)  to  using  the  ROI  process,  and  it  is 


27 


widely  used,  but  it  remains  challenging  for  many  organizations  due  to  cost  and  time,  lack 
of  skills,  faulty  needs  assessment,  fear,  discipline  and  planning,  and  false  assumption 
(Phillips,  2007).  If  applied  consistently  and  comprehensively,  ROI  processes  allow  an 
organization  to  invest  in  relevant  training  for  its  employees  while  improving  the  actual 
training  program.  Three  popular  methods  are  the  Kirkpatrick’s  four  levels  model, 
Schmidt’s  seven  steps  model,  and  Phillips’  five-level  ROI  framework. 
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III.  ANALYSIS  OF  THE  METHODOLOGIES 


A.  KIRKPATRICK’S  FOUR  LEVELS  MODEL 

The  Kirkpatrick  model  is  a  standard  model  used  to  determine  ROI.  The  simple 
four- level  approach  is  used  to  determine  the  effectiveness  of  training  programs.  However, 
criticism  of  the  model  centers  on  its  description  of  a  simple  taxonomy  instead  of  a  full, 
researchable  model.  It  is  also  criticized  because  evaluators  tend  to  skip  level  3  and  level  4 
(Pulichino,  2007). 

The  model  is  described  as  taxonomy  because  each  of  the  four  levels  are 
independent  of  the  others.  In  trying  to  evaluate  a  whole  training  program,  Kirkpatrick’s 
levels  are  unrelated  to  each  other,  and  each  level  requires  its  own  independent  evaluation. 
For  instance,  there  is  not  necessarily  a  correlation  between  the  results  of  one  level  and  the 
next.  Participants  can  react  to  the  training  in  a  favorable  way  (level  1),  yet  not  acquire  the 
intended  learning  or  skills  presented  (level  2;  Pulichino,  2007). 

Pulichino  (2007)  goes  on  to  critique  that  because  evaluators  must  evaluate 
training  across  four  distinct  and  different  criteria  in  the  Kirkpatrick  model,  there  is 
inherent  complexity  despite  the  “simple”  four  levels.  Evaluators  can  reach  contradictory 
conclusions  about  the  effectiveness  of  a  training  program.  Evaluators  may  also  force 
evaluation  of  a  program  using  all  four  levels,  when  not  all  trainings  need  to  be  evaluated 
at  all  levels  (Pulichino,  2007). 

Lacking  instruction  on  implementing  the  model,  evaluators  often  skip  levels  3  and 
4.  Difficulty,  complexity,  and  time  increase  with  evaluating  each  level,  which  contributes 
to  the  omission  of  higher  levels  of  evaluation.  For  evaluating  level  3,  difficulty  is 
identified  in  gathering  the  data,  designing  the  interview,  determining  the  right  people  to 
survey,  lack  of  time,  and  the  cost  and  complexity  involved  in  using  a  control  group. 
Training  professionals  can  struggle  in  how  to  relate  results  to  training  (Pulichino,  2007). 

Kirkpatrick  needs  to  include  additional  factors  and  variables  in  determining  the 
effectiveness  of  a  program.  For  instance,  in  level  1,  the  opportunity  to  apply  learned 
behavior  to  the  job  may  be  inhibited  by  the  work  environment.  If  the  work  environment 
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constrains  the  learning  or  does  not  provide  a  climate  that  encourages  change  (level  3),  a 
participant  could  react  favorably  to  training,  but  the  discouraging  work  environment  does 
not  support  change  (Sekowski,  2002). 

Additionally,  at  level  4,  Kirkpatrick  does  not  factor  in  the  direct  cost  of  training. 
Therefore,  salary  for  instructors,  supplies  and  materials,  and  the  trainees’  time  away  from 
work  are  not  factored  into  his  model  (Sekowski,  2002).  Other  variables  not  accounted  for 
include  the  level  of  knowledge  of  participants,  their  experience,  their  position  in  the 
organization,  the  knowledge  of  the  training  staff,  the  size  of  the  organization,  and  the 
type  of  industry.  The  consequences  of  omitting  these  factors  could  result  in  false 
assumptions  in  change  of  job  behavior,  opportunity  and  time  to  apply  the  learned 
behavior,  and  changes  that  may  have  impacted  the  market  (Pulichino,  2007). 

Because  of  the  vague  instructions  to  implement  Kirkpatrick’s  four  levels  model 
and  increased  complexity  in  evaluating  levels,  evaluators  are  not  using  the  right  metrics, 
not  using  the  right  data,  and  developing  superficial  and  subjective  evaluations.  The 
superficial  findings  can  lead  to  overgeneralizing  the  overall  program  and  lead  to  a 
misunderstanding  of  the  results  (Pulichino,  2007). 

B.  SCHMIDT’S  SEVEN  STEPS  MODEL 

Schmidt’s  seven  steps  model  for  evaluating  ROI  aims  to  expand  the  Kirkpatrick 
model  and  measure  the  value  of  training  to  the  organization  and  individual  objectively, 
quantitatively,  and  credibly.  Using  Kirkpatrick’s  four  levels  model,  the  Schmidt  model 
condenses  the  four  levels  into  three  and  adds  four  more  levels,  two  to  be  conducted 
before  training  and  two  after  training  (Schmidt,  2017). 

Schmidt  (2017),  in  step  1,  identifies  the  need  for  a  formal  training  advisory  group 
to  determine  the  training  needs.  The  group  would  be  made  up  of  supervisors  who  know 
their  subordinates’  strengths  and  weaknesses.  Knowing  their  people,  they  can  then 
determine  the  training  needs.  This  step  relies  on  a  supervisor/subordinate  history.  A 
unique  element  to  the  military  is  the  constant  rotation  of  personnel.  With  officer  tours 
typically  lasting  only  three  years,  and  with  staggered  rotation  dates  within  a 
command/office,  a  supervisor  in  the  military  may  be  challenged  to  effectively  identify 
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training  needs  based  on  personnel  knowledge  of  employees.  In  the  case  of  the  TWI 
program,  the  supervisor  following  the  fellowship  had  no  say  in  establishing  the  training 
objectives  and  typically  would  not  know  the  participant,  let  alone  the  participant’s 
weaknesses  that  the  TWI  program  might  help  them  overcome. 

Step  2  aims  to  link  training  and  performance  measures.  This  involves  both  HR 
and  training  personnel  to  develop  a  metric  or  scorecard.  The  issue  with  performance 
measurement  systems  is  that  they  must  be  effective.  If  a  performance  measurement 
system  is  not  effective,  the  training  metric  will  have  no  value  (Schmidt,  2017). 

Step  3  encompasses  Kirkpatrick’s  level  1,  or  response  to  training.  However, 
Kirkpatrick  relies  on  general  scorecards  to  evaluate  responses  to  training.  This  method  is 
only  helpful  at  times  when  the  training  participants  overwhelmingly  agree  to  something 
as  either  very  good  or  very  bad.  So,  Schmidt  (2017)  developed  specific,  direct  questions 
for  each  content  area.  By  requiring  participants  to  explain  how  the  training  will  impact 
the  job  or  provide  benefit,  the  supervisor  can  then  look  for  those  specific  elements  to 
change  in  job  performance.  Evaluating  the  responses  to  direct  questions,  such  as  the 
personal  and  professional  value  gained  from  the  training  or  areas  to  improve  the  training, 
can  provide  the  necessary  feedback  to  HR  and  training  to  determine  best  uses  for  training 
resources  (Schmidt,  2017). 

To  measure  learning  and  retention  of  information,  the  Schmidt  model  expands  on 
Kirkpatrick’s  level  2.  To  show  change  in  behavior  and  impact  to  business  after  training, 
Schmidt  follows  Kirkpatrick  with  a  pre-test  and  post-test.  However,  Schmidt  also 
includes  a  second  post-test,  weeks  after  training,  to  evaluate  retention  of  learned 
knowledge  and  skill.  The  second  post-test  is  meant  to  determine  (a)  “how,  where,  and  by 
how  much  their  own  productivity  or  efficiency  has  improved”  and  (b)  “where  has  the 
training  helped  improve  the  quality  of  their  work”  (Schmidt,  2017).  As  stated  for  the 
military,  after  completing  the  TWI  program,  participants  report  to  a  new  command. 
Stepping  into  a  new  job,  it  is  not  possible  to  identify  the  TWI  program  as  the  sole  factor 
of  performance  improvement.  There  is  no  base  for  comparison.  Nor  would  a  supervisor 
know  whether  the  behaviors  would  have  improved  without  TWI,  based  on  natural 

improvement  and  progression  in  a  military  career. 
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Step  5  for  Schmidt  (2017)  combines  level  3  and  level  4  from  the  Kirkpatrick 
model  to  measure  the  changed  behavior  of  the  participant  and  the  impact  to  business. 
Here  supervisors  evaluate  the  skills  and  performance  of  the  participant  both  before  and 
after  training.  Areas  to  look  for  as  improved  after  training  are  productivity  and  efficiency, 
quality  of  work,  new  skills  or  abilities,  and  linking  the  changed  behaviors  to  business 
objectives.  This  link  entails  putting  a  dollar  value  on  each  business  objective  expected 
from  the  training,  determining  what  the  business  should  pay  to  achieve  the  objective,  and 
then  determining  what  percent  is  attributed  to  training.  From  there,  supervisors  can 
determine  if  training  is  essential  to  meet  the  objective  (Schmidt,  2017). 

Again,  the  problem  with  this  step  for  the  military  is  lack  of  continuity  between 
both  supervisors  and  participants  in  the  TWI  program.  Supervisors  do  not  work  with  TWI 
participants  prior  to  their  training  experience.  Because  of  this,  the  supervisor  cannot 
determine  any  link  to  training  and  change  in  performance.  With  the  current  pipeline  for 
TWI  participants  and  the  subsequent  transfer  to  a  new  job  after  the  TWI  program,  this 
step  is  not  conducive  to  determining  ROI  for  TWI  participants. 

Schmidt’s  step  6  (2017),  capitalizes  on  weaknesses  identified  in  the  Kirkpatrick 
model.  In  this  step,  all  training  costs  are  measured.  The  challenge  here  is  to  determine  the 
cost  per  successful  participant.  Success  is  defined  as  changed  performance.  Numbers  can 
be  inflated  to  document  a  higher  return  and  be  decreased  to  lower  investment  costs. 
Therefore,  identifying  costs  needs  to  be  considered  and  applied  objectively.  This  step 
allows  for  the  cost  consideration  of  items  like  (a)  housing  and  travel  allowances,  (b) 
marketing  positive  impact  to  participants,  (c)  marketing  positive  career  impact  to  future 
participants,  and  (d)  calculating  the  opportunity  costs  of  participants  being  away  from  the 
day-to-day  work  for  training.  For  the  Navy,  this  is  the  opportunity  cost  of  being  away 
from  an  operational  fleet  for  12  months. 

It  is  at  step  7  that  Schmidt  delivers  a  full  training  report  card.  While  the  report 
does  not  communicate  everything  that  is  important  about  the  ROI  from  training,  it  does 
provide  participant  feedback,  data  from  post-tests  to  determine  what  was  learned  and 
retained,  changes  in  performance,  the  impact  on  the  business,  and  the  cost  of  a  successful 
participant  (Schmidt,  2017). 
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C.  PHILLIPS’  FIVE-LEVEL  ROI  FRAMEWORK 

Phillips’  five-levels  of  ROI  framework  takes  the  best  of  the  Kirkpatrick  model 
and  adds  a  fifth  level.  Phillips’  level  5  is  a  cost-benefit  analysis  using  the  classic  ROI 
formula.  In  addition  to  Kirkpatrick’s  levels  1-4,  Phillips  elaborates  on  the  importance  of 
data  collection,  isolating  the  effect  of  the  training  program,  converting  data  to  monetary 
values,  tabulating  program  cost  for  an  accurate  ROI  calculation,  and  identifying 
intangible  benefits  (Phillips,  2003).  Phillips  ensures  a  systematic  evaluation  of  the 
transfer  of  the  learned  skill  from  the  training  program  and  its  organizational  impact. 
Phillips  emphasizes  the  importance  of  simplicity  of  the  ROI  process,  but  recognizes  that 
truly  capturing  an  accurate  ROI  can  be  complex  (Phillips,  1997). 

The  value  added  from  Phillips’  level  5  is  the  ability  to  compare  the  monetary 
benefits  from  the  program  and  its  costs  (Chmieliewski  &  Phillips,  2002).  The  procedures 
for  measuring  the  investment,  as  well  as  indicators  of  return,  factor  in  both  soft  data  and 
hard  data.  Soft  data  includes  employee  attitudes,  the  organization  climate,  and  work 
habits.  Hard  data  is  the  output,  quality,  time,  and  cost.  The  challenge  lies  in  converting 
the  data,  especially  soft  data,  into  a  dollar  value.  If  the  value  of  data  becomes  too 
subjective,  the  resulting  values  lose  credibility.  With  enough  quality  data,  Phillips’  level 
5  provides  information  on  the  training  program,  including  whether  the  program  is  too 
costly,  what  the  savings  are,  and  what  costs  were  avoided  by  implementing  the  program 
(Chmieliewski  &  Phillips,  2002). 

D.  RECOMMENDED  METHODOLOGY 

The  military  manning  and  rotation  process  makes  choosing  one  model  for 
determining  ROI  difficult.  There  is  not  one  plug-and-play  methodology  that  fits  perfectly 
with  the  military  and  the  TWI  program.  However,  through  picking  and  choosing  elements 
from  various  ROI  models,  altering  the  Schmidt  seven  step  model,  which  already 
incorporates  both  Kirkpatrick’s  and  Phillips’  models,  is  the  recommended  methodology 
for  determining  the  TWI  program’ s  ROI. 
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1.  Step  1:  NAVSUP  Identifies  Objectives  for  Each  TWI  Location 

With  military  billets,  there  is  not  much  continuity  in  the  relationship  between 
supervisor  and  subordinate.  Both  are  typically  in  a  particular  billet  for  only  three  years, 
and  tours  between  the  two  generally  overlap,  creating  a  period  of  maybe  two  years  where 
the  supervisor  can  identify  areas  for  subordinate  improvement  and  assess  performance 
after  training.  Therefore,  a  training  advisory  team  as  Schmidt  suggests  is  not  conducive 
for  the  TWI  program.  Plus,  supervisors  do  not  know  their  subordinates  until  after 
completing  the  TWI  program. 

However,  the  Navy  Supply  Corps  has  already  laid  out  objectives  for  each 
participant  based  on  the  location  of  the  tour.  Prior  to  partnering  with  an  industry  partner, 
the  commander  of  NAVSUP  requests  permission  from  the  secretary  of  the  Navy 
(SECNAV)  to  establish  the  TWI  program.  For  each  company,  the  request  identifies 
learning  objectives  the  participant  is  expected  to  accomplish.  The  Navy  may  need  to 
refine  the  objectives  to  ensure  they  are  measurable  and  linked  to  business  objectives  of 
the  Navy  and/or  objectives  for  the  professional  growth  of  the  participant  (Schmidt,  2017). 

As  participants  provide  feedback  and  as  the  Navy’s  training  needs  change, 
objectives  should  change  as  well.  Additionally,  frequent  feedback  from  supervisors  in  the 
fleet  should  drive  program  objectives.  Once  a  new  participant  starts  the  program,  he 
should  have  a  copy  of  the  objectives.  The  TWI  partner  should  also  have  a  copy  of  the 
objectives.  Additionally,  during  the  participant’s  follow-on  utilization  tour,  his  supervisor 
should  have  a  copy  of  the  objectives. 

2.  Step  2:  Link  TWI  ROI  to  Existing  NAVSUP  Performance 
Measurements 

The  Navy  should  then  develop  a  scorecard  on  how  participants  meet  program 
objectives.  For  instance,  TWI  in  the  Supply  Corps  focuses  on  supply  functions.  A 
scorecard  should  define  and  rate  the  training  outcomes  based  on  supply  functions,  such  as 
supply  chain  management,  contracting  support,  financial  management,  quality  assurance, 
and  leadership.  A  performance  measure  of  supply  chain  knowledge  gained  through  the 
TWI  program  may  state,  “Applies  Fean  Six  Sigma  processes  from  the  corporate  sector  to 
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improve  government  processes.”  The  resulting  score  could  be  a  stoplight  color  code  of 
green,  yellow,  or  red. 


3.  Step  3:  Response  to  the  TWI  Program 

In  this  step,  the  TWI  participant  tells  NAVSUP  the  value  TWI  brings  to  the  Navy. 
Specific  questions  should  be  asked  to  determine  benefits,  value,  and  improvements  to  the 
program  (Schmidt,  2017). 

•  How  do  you  expect  to  use  what  you  learned  at  [FedEx]  in  the  Navy? 

•  How  did  the  training  contribute  to  your  qualities  as  an  officer? 

•  How  will  your  training  improve  your  quality  of  work  for  the  Navy? 

•  How  did  this  training  contribute  to  your  professional  goals? 

•  What  new  skill/knowledge  obtained  is  most  beneficial  to  the  Navy? 

•  What  aspect  of  TWI  was  least  valuable? 

Having  answers  to  questions  like  these  will  show  supervisors  where  to  look  for 
performance  improvement  and  impacts  to  the  Navy.  It  will  also  help  the  supervisor  with 
inputs  to  periodic  fitness  reports,  while  providing  the  participant  with  a  way  ahead  in 
using  his  new  skills  and  knowledge. 

4.  Step  4:  TWI  Learning  and  Retention 

This  step  creates  a  challenge  for  the  Navy.  Schmidt’s  intent  in  his  ROI  model  is  to 
conduct  a  pre-course  testing,  post-course  exam,  and  a  second  post-course  exam.  Because 
the  TWI  program  is  more  like  an  internship,  where  practical  experience  is  gained  versus 
training  for  a  specific  skill,  pre-  or  post-exams  do  not  make  sense. 

However,  step  3’s  questions  and  the  participant’s  response  to  the  TWI  program 
can  be  used  as  a  post  and  second  post  examination.  The  questions  can  be  asked  and 
recorded  prior  to  transferring  from  the  TWI  location  to  the  next  billet  location.  The 
questions  can  be  answered  again  4-6  months  later  during  the  participant’s  utilization 
tour.  The  4-6-month  period  of  settling  in  to  a  new  billet,  understanding  the  job,  and  re¬ 
acclimating  to  the  Navy  will  help  the  participant  reflect  on  how  the  TWI  experience  can 
impact  the  Navy.  The  participant  may  even  notice  new  ways  in  which  he  is  a  more 

productive,  efficient  and  effective  officer  than  realized  after  the  initial  completion  of  the 
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TWI  program.  A  significant  question  to  answer  might  be,  “What  am  I  doing  now  for  the 
Navy  that  I  could  not  do  before?”  (Schmidt,  2017). 

5.  Step  5:  Navy  Impact 

According  to  Schmidt  (2017),  “Training  has  value  to  the  company  or  organization 
only  when  it  leads  to  changes  in  behavior  that  meet  business  objectives.”  NAVSUP  has 
implemented  the  program  to  “provide  training  and  development  of  skills  in  private  sector 
procedures,  and  practices  that  are  not  available  through  existing  military,  advanced 
civilian  schooling  programs  or  other  established  training  and  education  programs” 
(Heinrich,  2012).  These  skills  and  knowledge  gained  provide  a  “return”  for  the  Navy.  In 
meeting  these  objectives,  NAVSUP  has  to  put  a  value  on  each  objective.  If  the  TWI 
participant  meets  an  objective,  how  much  can  be  attributed  to  the  TWI  experience? 
Answering  this  question  will  determine  “whether  training  is  essential  to  reaching  the 
objective,  or  instead  one  of  many  contributing  factors”  (Schmidt,  2017). 

6.  Step  6:  Measure  All  Training  Costs  and  Benefits 

This  step  pulls  from  the  Phillips  model  and  is  used  for  determining  the  program’s 
benefits  and  costs.  The  formula  to  determine  an  ROI  percentage  is  Net  Program 
Benefits/Program  Costs  x  100.  It  is  determining  the  tangibles  and  intangibles  that  create 
the  challenge.  Tangibles  like  salary  and  travel  can  be  easy  to  identify.  However, 
intangibles  like  increased  job  satisfaction  or  increased  commitment  to  the  Navy  are  more 
subjective  and  difficult  to  put  a  monetary  value  on  (Phillips,  2003). 

7.  Step  7:  Full  Training  Report 

Bringing  all  the  information  together  will  provide  a  comprehensive  view  as  to  the 
return  the  Navy  gets  on  the  TWI  program.  Identifying  objectives,  creating  a  scorecard 
based  on  performance,  determining  how  participants  apply  what  was  learned,  placing  a 
value  on  meeting  objectives,  and  determining  the  cost-benefit  analysis  will  provide  an 
overall  ROI  for  the  TWI  program  (Schmidt,  2017). 
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E.  CHAPTER  SUMMARY 


Determining  ROI  is  always  a  topic  for  debate.  Since  the  1950s,  people  have  been 
trying  to  create  models  for  determining  ROI.  And  as  budgets  are  cut  in  the  government 
and  training  dollars  are  pinched,  validating  the  ROI  for  training  has  become  more 
important.  Because  there  is  no  one  perfect  model  for  every  training,  it  is  necessary  to 
piecemeal  elements  of  various  models  to  make  a  sound  model  to  best  meet  individual 
trainings.  In  the  case  of  the  TWI  program,  elements  of  Kirkpatrick,  Schmidt,  and  Phillips 
were  utilized  to  best  present  a  model  for  the  Navy  to  use. 

The  following  chapter  takes  the  elements  of  Kirkpatrick,  Schmidt,  and  Phillips’ 
methodologies  and  creates  a  comprehensive  metric  to  measure  the  ROI  for  the  TWI 
program.  The  metric  takes  current  capability  gaps,  program  objectives,  and  developed 
forms  to  provide  NAVSUP  with  a  product  to  determine  the  ROI  for  the  TWI  program.  A 
full  ROI  cannot  be  measured  until  NAVSUP  implements  the  recommendations  and  tracks 
new  selected  TWI  participants  from  cradle  to  grave.  However,  all  attempts  are  made  to 
explain  the  full  data  collection  process  and  how  to  interpret  results. 
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IV.  APPLYING  THE  METHODOLOGY 


Choosing  one  methodology  to  use  on  the  TWI  program  proves  challenging  and 
unfitting  due  to  the  unique  nature  of  the  government’s  way  of  doing  business  from  the 
commercial  industry.  Instead,  this  project  uses  elements  from  the  three  ROI  models 
discussed  with  Schmidt’s  seven  steps  model  as  its  analytical  framework.  Figure  6  is  the 
recommended  Navy  TWI  ROI  Model  which  incorporates  the  elements  from  the  three 
ROI  models  discussed  in  Chapter  II. 


Is  the  report  useful  for  the  audience? 


Step  7:  Full 
Training  Report 


Were  all  costs  and  benefits  to  include 
intangible  benefits  considered? 


Step  6:  Measure  All 
Costs  and  Benefits 


k 


How  was  the  knowledge  gained  used 
to  impact  the  Navy? 


Step  5:  Navy  Impact 

Was  knowledge  gained  and  retained  to 

use  in  their  utilization  tour? 

) - 

k 

Step  4:  TWI  Learning  and  Retention 


Were  participants  motivated  and  viewed 
TWI  as  relevant  and  value  added? 


Step  3 :  Response  to  the  TWI 


How  does  NAVSUP  measure  performance 
and  can  we  use  it  to  determine  ROI? 


Step  2:  Link  TWI  ROI  to  Existing  NAVSUP 
Performance  Measurements 


Is  TWI  objectives  linked  to 
NAVSUP’s  objectives? 


Step  1 :  NAVSUP  Identify  Objectives  for  each  TWI  Location 


Figure  6.  Navy  TWI  ROI  Model.  Adapted  from  Schmidt  (2017). 

A.  NAVY  TWI  ROI  MODEL 

As  discussed  in  Chapter  II,  simply  going  straight  to  the  ROI  formula  is  not 
sufficient  for  the  measurement  of  a  leadership  training  skill  of  this  type.  A  holistic 
approach  is  necessary  to  truly  evaluate  a  program.  NAVSUP,  the  owner  of  the  Navy’s 
TWI  program,  is  in  the  best  position  to  utilize  the  Navy  TWI  ROI  Model.  The  following 
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provides  an  example  to  implement  the  Navy  TWI  ROI  Model  in  order  to  evaluate  the 
entire  TWI  program  while  realizing  the  ROI  in  step  6. 

1.  Step  1:  NAVSUP  Identifies  Objectives  for  Each  TWI  Location 

Three  steps  help  identify  the  objectives  for  each  location. 

a.  Identify  the  skill  set  needed  to  fulfill  an  existing  Military  Component  or 
for  desired  future  capability  as  stated  in  DOD  Instruction  1322.06 
(2007). 

The  U.S.  Navy  Supply  Corps  aims  to  continuously  align  itself  with  the  fleet  to 
ensure  products  and  services  are  delivered  to  the  warfighter  to  meet  the  readiness  needs 
of  all  its  naval  customers.  To  accomplish  this  objective,  NAVSUP  has  established  and 
published  four  strategic  goals  in  the  NAVSUP  Strategic  Plan  2013-2017  (NAVSUP, 
n.d.-b).  These  four  goals  are: 

•  Create  and  sustain  a  working  environment  that  fosters  teamwork  and 
collaboration,  rewards  innovation,  and  provides  the  tools,  resources,  and 
developmental  opportunities  employees  need  to  be  effective. 

•  Drive  unity  of  effort  across  the  Naval  support  network  by  strengthening 
relationships,  optimizing  processes,  and  focusing  on  customer  outcomes. 

•  Refine  internal  business  processes  to  reduce  operating  costs  and  ensure 
compliance  while  enhancing  support  to  our  customers. 

•  Create  and  sustain  an  information  environment  that  leverages  technology 
to  deliver  greater  transparency,  facilitate  information  data  sharing,  and 
enable  better  decision  making,  (pp.  3-4) 

The  capabilities  gaps  identified  according  to  NAVSUP  in  requesting  the  TWI 
partnerships  (Stone,  2004;  Lyden,  2009;  Heinrich,  2012)  are  as  follows: 

•  Exposure,  training,  and  skills  in  industry  innovative  business  approaches, 
management  practices,  and  corporate  organizational  culture 

•  Exposure  to  industry  senior  and  executive  level  decision-making  processes 

•  Develop  skills  in  private  sector  procedures  and  best  practices 

•  Understand  strategic  and  service-level  linkages  with  public  policy, 
corporate  and  non-governmental  organizations 

•  Ability  to  embrace  changes  and  other  cultures  in  a  complex  national 
environment 
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b.  Identify  billets  the  TWI participant  will  be  expected  to  fulfill  upon  TWI 
completion  as  required  in  DOD  Instruction  1322.06  (2007). 

In  each  request  for  authorization  of  the  TWI  program  letter  (Stone,  2004;  Lyden, 
2009;  Heinrich,  2012),  NAVSUP  identifies  measurable  objectives  the  TWI  participant  is 
to  attain.  Additionally,  the  letters  provided  several  examples  of  billets  for  utilization  tours 
upon  completion  of  the  TWI  program.  Tables  4  through  7  display  the  billets. 

c.  Identify  the  measurable  training  objectives  for  each  location  keeping  in 
mind  to  link  it  to  NAVSUP’s  objectives 

Tables  4  through  7  display  the  objectives  for  each  location  using  the  MOU 
between  the  Navy  and  the  companies. 


Table  4.  Identifying  Objectives — The  Home  Depot.  Adapted  from 

NAVSUP  (n.d.-b,  pp.  3-4),  Stone  (2004). 


STEP  1:  Identify  objectives  for  each  TWI  location 

The  Home  Depot 

NAVSUP  Strategic  Plan  2013-2017 

Capability  Gaps 

Measurable  Objectives 

Billets  (not  limited  to) 

Organization  Goals 

Goal  1.  World  Class  Workplace  - 

1)  Exposure,  training,  and 

Operations: 

1)  Weapon  System 

Add  the  command 

Create  and  sustain  a  working 

skills  in  industry 

-  Learn  the  store/business  operations  within  a 

Manager 

objectives  that  links 

environment  that  fosters  teamwork 

innovative  business 

Home  Depot  store 

between  the  measurable 

and  collaboration,  rewards 

approaches,  management 

-  Learn  operations  business  and  key  processes 

2)  Equipment  Manager 

objectives  learned  from 

innovation,  and  provides  the  tools, 

practices,  and  corporate 

-  Participate  in  Senior  Vice  president  (SVP) 

Support  and  Submarine 

industry  to  NAVSUP 

resources,  and  developmental 

organizational  culture 

Operations  Leadership  Team 

Program  Support 

Strategic  goals. 

opportunities  employees  need  to  be 

-  Participate  in  development  and 

effective. 

2)  Exposure  to  industry 

implementation  of  strategic  operations 

3)  Operations  Officer 

1)  NAVSUP  WSS 

senior  and  executive 

business  initiative 

Philadelphia 

Goal  2.  Unitv  of  Effort  -  Drive  unitv  of 

level  decision-making 

4)  Special  Project 

effort  across  the  Naval  support 

processess 

Human  Resources: 

Manager 

2)  NAVSUP  WSS 

network  by  strengthening 

-  Learn  key  HR  processes  (e.g.  performance 

Mechanicsburg 

relationships,  optimizing  processes, 

3)  Develop  skills  in 

management,  HR  review,  360  Feedback 

5)  Joint  Weapon  System 

and  focusing  on  customer  outcomes. 

private  sector  procedures 

process) 

Integrator 

3)  Defense  Logistics 

and  best  practices 

-  Learn  key  HR  programs  (e.g.  Store 

Agency 

Goal  3.  Effective,  Efficient 

Leadership  Programs,  Business  Leadership 

6)  Joint  Marine  Weapon 

Performance -Refine  internal 

4)  Understand  strategic 

Programs) 

System 

4)  Defense  Supply  Center 

business  processes  to  reduce 

and  service-level  linkages 

-  Shadow  assignment  with  Chief  Executive 

Richmond 

operating  costs  and  ensure 

with  public  policy, 

Officer  (CEO),  Store  Human  Resource 

compliance  while  enhancing  support 

corporate  and  non- 

Manager  (HRM),  etc. 

5)  Defense  Supply  Center 

to  our  customers. 

governmental 

organizations 

Merchandising: 

Columbus 

Goal  4.  Data  Driven  Decision  Making- 

-  Learn  merchandising  business  and  key 

6)  Defense  Support 

Create  and  sustain  an  information 

5)  Ability  to  embrace 

processes 

Center  Philadelphia 

environment  that  leverages 

changes  and  other 

-  Participate  in  Mechandising  Leadership 

technology  to  deliver  greater 

cultures  in  a  complex 

Team 

transparency,  facilitate  information 

national  environment 

-  Participate  in  development  and 

data  sharing,  and  enable  better 

implementation  of  strategic  merchandising 

decision  making. 

business  initiative 

-  Participate  in  Merchandising  Leadership 
Program  Career  Forum  and  Orientation 
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Table  5.  Identifying  Objectives — FedEx.  Adapted  from 
NAVSUP  (n.d.-b,  pp.  3-4),  Lyden  (2009). 


STEP  1:  Identify  objectives  for  each  TWI  location 

Federal  Express  Corporation 

NAVSUP  Strategic  Plan  2013-2017 

Capability  Gaps 

Measurable  Objectives 

Billets  (not  limited  to) 

Organization  Goals 

Goal  1.  World  Class  Workplace  - 

1)  Exposure,  training,  and 

Supply  Chain  Management: 

1)  Weapon  System 

Add  the  command 

Create  and  sustain  a  working 

skills  in  industry 

-  Learn  FedEx  Supply  chain  management  and 

Manager 

objectives  that  links 

environment  that  fosters  teamwork 

innovative  business 

how  it  is  utilized 

between  the  measurable 

and  collaboration,  rewards 

approaches,  management 

-  Learn  the  technological  application  involved 

2)  Equipment  Manager 

objectives  learned  from 

innovation,  and  provides  the  tools, 

practices,  and  corporate 

and  their  purpose 

Support  and  Submarine 

industry  to  NAVSUP 

resources,  and  developmental 
opportunities  employees  need  to  be 

organizational  culture 

Contract  Management: 

Program  Support 

Strategic  goals. 

effective. 

2)  Exposure  to  industry 

-  Company  orientation  to  support  contract 

3)  Operations  Officer 

1)  NAVSUP  WSS 

senior  and  executive 

managment 

Philadelphia 

Goal  2.  Unitv  of  Effort  -  Drive  unitv  of 

level  decision-making 

-  Company's  structure  to  influence  critical 

4)  Helicopter  Production, 

effort  across  the  Naval  support 

processess 

decision  point 

Customer  Support  Divsion 

2)  NAVSUP  WSS 

network  by  strengthening 
relationships,  optimizing  processes, 

3)  Develop  skills  in 

Acquisition  Pre-Award  Activities: 

5)  Joint  Customer  Service 

Mechanicsburg 

and  focusing  on  customer  outcomes. 

private  sector  procedures 

-  Business  strategy 

Team  Leader 

3)  Defense  Logistics 

and  best  practices 

-  Steps/responsibilities  involving  proposal 

Agency 

Goal  3.  Effective,  Efficient 

preparation,  contract  negotiation  and  award 

Performance -Refine  internal 

4)  Understand  strategic 

4)  Defense  Supply  Center 

business  processes  to  reduce 

and  service-level  linkages 

Award  Activities: 

Richmond 

operating  costs  and  ensure 

with  public  policy, 

-  Contract  administration  procedures  and 

compliance  while  enhancing  support 

corporate  and  non- 

responsibilities 

5)  Defense  Supply  Center 

to  our  customers. 

governmental 

organizations 

Pricing: 

Columbus 

Goal  4.  Data  Driven  Decision  Making- 

-  Learn  FedEx  cost  accounting  system  in 

Create  and  sustain  an  information 

5)  Ability  to  embrace 

relation  to  government  and  commercial 

environment  that  leverages 

changes  and  other 

business 

technology  to  deliver  greater 

cultures  in  a  complex 

-  Learn  FedEx  pricing  and  proposal 

transparency,  facilitate  information 

national  environment 

preparation  precidure 

data  sharing,  and  enable  better 
decision  making. 

Commercial  Practice: 

-  Learn  different  business  strategy  for 
government  and  commercial  customer 

Total  Quality: 

-  Learn  FedEx  approaches  to  total  quality 
philosophy 
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Table  6.  Identifying  Objectives — ExxonMobil.  Adapted  from 

NAVSUP  (n.d.-b,  pp.  3-4),  Heinrich  (2012). 


STEP  1:  Identify  objectives  for  each  TWI  location 

ExxonMobil  Fuels  Marketing  Company 

NAVSUP  Strategic  Plan  2013-2017 

Capability  Gaps 

Measurable  Objectives 

Billets  (not  limited  to) 

Organization  Goals 

Goal  1.  World  Class  Workplace  - 

1)  Exposure,  training,  and 

Supply  Operations: 

1)  Fuel  Depot 

Add  the  command 

Create  and  sustain  a  working 

skills  in  industry 

-  Perform  in  various  key  position  in  company's 

objectives  that  links 

environment  that  fosters  teamwork 

innovative  business 

operation  to  learn  functions  and 

2)  Aircraft  Fueling  Officer 

between  the  measurable 

and  collaboration,  rewards 

approaches,  management 

responsibilities 

objectives  learned  from 

innovation,  and  provides  the  tools, 

practices,  and  corporate 

3)  General  Supply 

industry  to  NAVSUP 

resources,  and  developmental 

organizational  culture 

Refinery  process: 

Strategic  goals. 

opportunities  employees  need  to  be 

-  Learn  various  procidure  essential  to  the 

4)  Fuel  Depot 

effective. 

2)  Exposure  to  industry 

refinery  process 

1)  NAVSUP  FLC  Diego 

senior  and  executive 

5)  Fuel  Logistics  Planner 

Garcia 

Goal  2.  Unity  of  Effort  -  Drive  unitv  of 

level  decision-making 

Downstream  and  Chemical  Safety,  Health,  & 

effort  across  the  Naval  support 

processess 

Environment  (SH&E): 

6)  Fuel  Depot 

2)  NAVSUP  FLC  Sigonella 

network  by  strengthening 

-  Participate  in  EXXONMOBILE's  Operation 

relationships,  optimizing  processes, 

3)  Develop  skills  in 

Integrity  Managment  System  (OIMS) 

3)  NAVSUP  FLC  Souda  Bay 

and  focusing  on  customer  outcomes. 

private  sector  procedures 

Assessments  to  familiarize  with  process  of 

and  best  practices 

continuous  improvement  and  streamline 

4)  NAVSUP  FLC  Rota 

Goal  3.  Effective,  Efficient 

operations 

Performance -Refine  internal 

4)  Understand  strategic 

-  Learn  trend  analysis  of  safety  statistics, 

5)  COMLOG  WESTPAC 

business  processes  to  reduce 

and  service-level  linkages 

assessment  results,  etc. 

operating  costs  and  ensure 

with  public  policy, 

6)  NAVSUP  FLC  Pearl 

compliance  while  enhancing  support 

corporate  and  non- 

General  Aviation: 

Harbor 

to  our  customers. 

governmental 

-  Design  and  operate  fixed  and  mobile 

organizations 

aviation  refueling  facilities  and  equipments 

Goal  4.  Data  Driven  Decision  Making- 

-  Supply  and  distribution  of  Jet  A  and  Avgas  to 

Create  and  sustain  an  information 

5)  Ability  to  embrace 

customers 

environment  that  leverages 

changes  and  other 

-  Conductinging  process  and  procedure 

technology  to  deliver  greater 

cultures  in  a  complex 

testing 

transparency,  facilitate  information 
data  sharing,  and  enable  better 

national  environment 

SeaRiver: 

decision  making. 

-  Manage  receipt  and  issue  operations  from 
inland  and  ocean-going  maritime  operations 

-  Coordinate  logistics  involved  with  moving 
various  petroleum  products 

-  Understand  and  improve  emergency 
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Table  7.  Identifying  Objectives — Starbucks.  Adapted  from  NAVSUP 
(n.d.-b,  pp.  3-4),  Lyden  (2009),  Stone  (2004). 


STEP  1:  Identify  objectives  for  each  TWI  location 

Starbucks 

NAVSUP  Strategic  Plan  2013-2017 

Capability  Gaps 

Measurable  Objectives 

Billets  (not  limited  to) 

Organization  Goals 

Goal  1.  World  Class  Workplace  - 

1)  Exposure,  training,  and 

Starbucks  Immersion: 

Unavailable,  but  should 

Unavailable,  but  should 

Create  and  sustain  a  working 

skills  in  industry 

-  Basic  understanding  of  Starbucks  Coffee 

be  similarto  Home  Depot 

be  similarto  Home  Depot 

environment  that  fosters  teamwork 

innovative  business 

Company 

and  FedEx 

and  FedEx 

and  collaboration,  rewards 

approaches,  management 

-  Familiarization  of  Starbucks  Supply  Chain 

innovation,  and  provides  the  tools, 

practices,  and  corporate 

Operations 

resources,  and  developmental 
opportunities  employees  need  to  be 

organizational  culture 

Supplly  Chain  orientation: 

effective. 

2)  Exposure  to  industry 

-  Familiarize  with  multiple  groups  and  their 

senior  and  executive 

impact  on  day-to-day  supply  chain  operation 

Goal  2.  Unitv  of  Effort  -  Drive  unitv  of 

level  decision-making 

and  strategy 

effort  across  the  Naval  support 

processess 

-  Understand  Starbucks  operational  functions 

network  by  strengthening 
relationships,  optimizing  processes, 

3)  Develop  skills  in 

and  scope  of  project 

and  focusing  on  customer  outcomes. 

private  sector  procedures 

Director  Orientation  and  project  work: 

and  best  practices 

-  Learn  Supply  Chain  Operations  at  director- 

Goal  3.  Effective,  Efficient 

level 

Performance -Refine  internal 

4)  Understand  strategic 

-  Understand  higher-level  processes,  and 

business  processes  to  reduce 

and  service-level  linkages 

cross-functional  project  work  with  category, 

operating  costs  and  ensure 

with  public  policy, 

retail  and  other  business  units  to  include 

compliance  while  enhancing  support 

corporate  and  non- 

international  businesses 

to  our  customers. 

governmental 

organizations 

Executive  Orientations: 

Goal  4.  Data  Driven  Decision  Making- 

-  Develop  an  understanding  of  supply  chain  at 

Create  and  sustain  an  information 

5)  Ability  to  embrace 

senior  leadersip  team  strategy  and  its 

environment  that  leverages 

changes  and  other 

relationship  within  the  framework  of 

technology  to  deliver  greater 

cultures  in  a  complex 

Starbucks  Coffee  Company. 

transparency,  facilitate  information 
data  sharing,  and  enable  better 
decision  making. 

national  environment 
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2.  Step  2:  Link  TWI  ROI  to  Existing  NAVSUP  Performance 
Measurements 

It  is  challenging  to  link  the  TWI  ROI  to  specific  existing  NAVSUP  performance 
metrics.  Each  NAVSUP  organization  uses  different  performance  metric.  For  the  scope  of 
this  project,  the  time  constraint  does  not  allow  analysis  of  all  possible  performance  metric 
of  all  possible  utilization  organizations.  Instead,  this  project  links  the  TWI  ROI  to 
NAVSUP’s  four  goals  as  stated  in  the  Strategic  Plan  2013-2017  (NAVSUP,  n.d.-b). 

a.  Hard  ROI 

Hard  ROI  is  the  increase  in  productivity  the  TWI  participant  brings  back  to  the 
Navy  after  completing  the  program.  The  costs  and  benefits  calculation  is  detailed  in 
step  6.  Hard  ROI  is  also  the  cost  saving  TWI  participant  may  contribute  to  while  at  their 
TWI  utilization  tour.  An  example  is  leading  a  project  that  improved  an  existing  process 
or  created  a  new  process  which  resulted  in  cost  saving  to  the  Navy.  This  data  was  not 
measured  for  this  MBA  project,  but  a  form  has  been  created  for  future  collection  of  this 
data. 


b.  Soft  ROI 

Soft  ROI  is  the  intangible  benefit  and  in  this  case,  a  more  important  benefit  than 
the  hard  ROI.  TWI  participants  will  obtain  skills  that  help  meet  NAVSUP’s  goals  as 
stated  in  the  NAVSUP  Strategic  Plan  2013-2017.  Figure  7  illustrates  the  link.  It  displays 
the  capability  gaps  identified  from  the  NAVSUP  Strategic  Plan  (NAVSUP,  n.d.-b).  These 
gaps  cannot  be  satisfied  with  “existing  military  or  advanced  civilian  schooling  programs 
or  other  established  training  and  education  programs”  (Heinrich,  2012).  The  TWI 
objectives  at  each  location  are  identified  to  ensure  fulfillment  of  existing  capability  gaps. 
The  billets  are  identified  to  best  use  TWI  participants’  newly  learned  skills.  In  performing 
their  duties  at  the  utilization  tour,  TWI  participants  meet  their  command’s  objectives, 
which  are  linked  to  NAVSUP’s  objectives.  As  a  result  of  this  link,  TWI  participants  meet 
NAVSUP’s  objectives  and  fulfilled  identified  capability  gaps,  which  is  arguably  the 
biggest  benefit  for  the  Navy. 
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Figure  7.  Link  between  NAVSUP  Objectives  and  TWI  Objectives 

Table  8  is  the  Data  Collection  Plan.  It  is  an  overview  of  what  objective(s)  to 
accomplish  during  each  stage  of  the  learning  process,  what  method  of  data  collection  will 
be  used  to  meet  the  objective(s),  when  the  data  collection  will  happen,  and  who  will  be 
responsible  for  the  data  collection. 


Table  8.  Data  Collection  Plan.  Adapted  from  Phillips  (1997,  p.  36). 


Data  Collection  Plan 

Program:  TWI  Responsibility:  Date: 

Level 

Objective  (s) 

Evaluation  Method 

Timing 

Responsibilties 

1 

Reaction,  Satisfaction  and  Planned 

Actions 

1)  Positive  Reaction 

2)  Recommended  Improvements 

3)  Action  Items 

1-3)  Initial  Program 
Evaluation  Form 

1-3)  Upon  completion  of  TWI  tour,  before  reporting  to  utilization 

1-3)  Sponsoring  company 
representative.  Email  a  copy  to 
PERS-4412. 

2 

Learning 

1)  Industry  innovative  business  approaches,  management 
practices,  and  corporate  organizational  culture 

2)  Industry  senior  and  executive  level  decision-making 
processes 

3)  Skills  in  private  sector  procedures  and  best  practices 

4)  Strategic  and  service-level  linkages  with  public  policy, 
corporate  and  non-governmental  organizations 

5)  Ability  to  embrace  changes  and  other  cultures  in  a  complex 
national  environment 

1-5)  Skill  Practice 

1-5)  During  TWI  tour 

1-5)  TWI  participants  and 
Sponsoring  company 
representative. 

3 

Job  Application 

1)  Meet  NAVSUP's  objecives 

2)  Use  of  Skills 

3)  Frequency  of  Skill  Use 

4)  Barriers 

1)  Scorecard 

2-4)  Post  Program 
Evaluation  Form  (4-6 
months) 

1)  Upon  completion  of  TWI  tour,  before  reporting  to  utilization 

2-4)  4-6  months  after  working  at  utilization  tour 

1)  Sponsoring  company 
representative.  Email  a  copy  to 
PERS-4412. 

2-4)  PERS-4412 

4 

Business  Results 

1)  Cost  saving  to  the  Navy 

2)  Fasterand  more  efficient  ways  to  conduct  business 

3)  Increased  customer  satisfaction 

1-3)  Capture  Navy 

Impacts  Data 
(performance 
monitoring) 

1-3)  One  to  two  years  after  working  at  utilization  tour 

1-3)  PERS=4412  and  TWI 
paticipants 
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In  the  Data  Collection  Plan,  upon  completion  of  the  TWI  tour  and  before 
reporting  to  the  utilization  tour,  the  participant  will  provide  information  to  the  scorecard 
and  the  initial  program  evaluation  form.  This  will  be  administered  by  the  partnering 
company’s  representative.  The  scorecard  is  a  way  to  assess  whether  the  participant  is 
meeting  the  objectives  set  out  by  the  program  and  how  each  of  the  objectives  are  linked 
to  NAVSUP’s  objectives.  Table  9  is  an  example  of  the  scorecard. 


Table  9.  TWI  Scorecard.  Adapted  from  NAVSUP  (n.d.-b,  pp.  3-4). 


TWI  SCORECARD 

NAVSUP  Strategic  Plan  2013-2017 

Objectives  for  [Home  Depot] 

YES 

NO 

Overall  Score 

Goal  1.  World  Class  Workplace  - 

Supply  Chain 

GREEN 

YELLOW 

Create  and  sustain  a  working 
environment  that  fosters 

teamwork  and  collaboration, 
rewards  innovation,  and  provides 
the  tools,  resources,  and 
developmental  opportunities 
employees  need  to  be  effective. 

Goal  2.  Unity  of  Effort  -  Drive 
unity  of  effort  across  the  Naval 

G3:  Know  the  production  schedule 

Met  70%  or 

above  of  the 
objectives 

Met  40-69%  of 
the  objectives 

Met  39%  or  less 
of  the  objectives 

G3:  Understands  inventory  management 

G2:  Can  coordinate  with  third  party  transporters 

G3:  Can  manage  inventory  at  inventory  control  point 

G3:  Know  how  to  evaluate  work  processes  on  the  line 

G4:  Can  identify  technology  applications  and  purposes  to  improve  supply  chain  management 

Note:  Add  relevant  objective  and  tz 

ke  away 
calculate  the 

G4:  Know  the  benefits  and  problems  the  company  faces  with  new  software/management  programs 

irrelevance  objective  as  needed.  Tc 

G3:  Know  how  companies  utilize  strategic  sourcing 

relationships,  optimizing 
processes,  and  focusing  on 

customer  outcomes. 

Goal  3.  Effective,  Efficient 

G3:  Know  how  the  use  of  strategic  sourcing  affects  the  bottom  line 

Overall  Score  =  (#  yes /Total  #  of  objectives)  x  100 

G3:  Know  the  key  processes  in  the  operation 

Quality  Assurance 

G2  =  NAVSUP  Goal  2 

G3:  Can  test  for  quality  of  product 

G3  =  NAVSUP  Goal  3 

Performance  -  Refine  internal 

G3:  Know  the  inspection  techniques  used  by  the  company 

G4  =  NAVSUP  Goal  4 

business  processes  to  reduce 
operating  costs  and  ensure 
compliance  while  enhancing 
support  to  our  customers. 

Goal  4.  Data  Driven  Decision 

Making -Create  and  sustain  an 

information  environment  that 
leverages  technology  to  deliver 
greater  transparency,  facilitate 
information  data  sharing,  and 
enable  better  decision  making. 

Fuel 

G3:  Understands  the  environmental  factors  to  refinery  process 

G3:  Can  assess  fuel  testing  procedures 

G2:  Understands  the  operation  of  aviation  refueling  facilities 

G3:  Can  conduct  process  and  procedure  testing  to  ensure  safe  practices 

G3:  Can  coordinate  movement  of  fuel  assets 

Safety 

G3:  Improve  upon  emergency  response  plans 

Contract  Management 

G3:  Improve  organization's  structure  to  optimize  acquisition  function 

G2:  Know  the  types  of  contracts  used  in  procurement  and  its  effects  on  the  contract  choice 

G2:  Know  different  negotiation  strategies  used  by  industry 

G2:  Know  how  industry  carries  out  contract  administration 

G2:  Know  how  industry  manage  subcontractors 

G2:  Know  how  business  strategy  changes  when  dealing  with  the  government  versus  commercial 
customers 

G2:  Know  the  techniques  used  to  incentivize  contracts 

Process  improvement 

G3:  Know  how  Lean/  Six  Sigma  is  utilized  and  incorporated  into  business 

G3:  Know  different  improvement  initiatives  companies  are  implementing 

Gl:  Know  what  processes  companies  used  to  develop  leadership 

Gl:  Know  how  companies  instill  strategic  and  operating  excellence 

TWI  participants  may  also  obtain  skills  that  contribute  to  their  personal  growth  as 
naval  officers.  In  theory,  there  should  be  an  improvement  in  the  participant’s  Fitness 
Report  (form  NAVPERS  1610/2  11-11),  specifically  in  Block  33  (Professional 
Expertise),  Block  36  (Teamwork),  Block  37  (Mission  Accomplishment  and  Initiative) 
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and  Block  38  (Leadership).  Currently,  obtaining  this  data  is  challenging,  as  is  isolating 
the  effect  of  the  program  due  to  so  many  factors  contributing  to  an  individual’s  general 
performance  evaluation.  However,  this  can  be  measured  if  the  TWI  track  is  changed  to 
have  more  continuity  between  the  participant  and  the  reporting  senior.  As  illustrated  in 
Figure  8,  a  selected  TWI  participant  reports  to  a  utilization  tour  for  one  year,  attends  the 
TWI  program,  and  reports  back  to  the  utilization  tour  command.  The  supervisor  of  the 
individual  will  see  the  difference  from  before  and  after  the  TWI  tour. 


Current  TWI  Process 

Selected 

1  year  TWI  tour 

Year  1  Utilization  Tour 

Year  2  Utilization  Tour 

Year  3  Utilization  Tour 

Future  TWI  Process 

Selected 

Year  1  Utilization  Tour 
Performance  Baseline 

1  year  TWI  tour 

Year  2  Utilization  Tour 

Year  3  Utilization  Tour 

_ J 

i 

A  total  of  4  years  at  one  command.  Command  should 
expect  returns  from  participants  at  Year  2  and  3  of 
utilization  tour. 

Figure  8.  Current  and  Future  TWI  Process 


3.  Step  3:  Response  to  the  TWI  Program 

Sponsoring  organizations  (e.g.,  Starbucks,  FedEx)  should  require  participants  to 
fill  out  the  initial  post  program  evaluation  (Table  10)  before  departure.  The  experience  of 
the  participants  will  still  be  fresh  on  their  minds  and  the  sponsoring  organization,  along 
with  NAVSUP,  will  be  able  to  make  immediate  improvement  to  the  program  using  the 
participant’s  feedback.  The  purpose  of  this  evaluation  is  to  capture  the  valuable  initial 
response  to  the  training  program  and  to  help  in  assessing  whether  the  TWI  program 
fulfills  the  five  identified  capability  gaps.  Completing  the  post  program  evaluation  should 
be  mandated  in  the  Memorandum  of  Understanding  (MOU)  between  the  Navy  and  the 
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partnering  company  to  institutionalize  the  process.  The  initial  post-program  evaluation 
can  be  used  to  improve  the  program  experience  for  future  participants  and  to  use  in  the 
overall  ROI  measurement  consideration.  The  following  tables  are  examples  of  a  TWI 
Program  Evaluation  form  (Table  10)  and  its  interpretation  (Table  11). 
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Table  10.  TWI  Program  Evaluation 


TWI  Program  Evaluation 


Please  help  us  evaluate  the  Training  with  Industry  (TWI)  program  by  answering  the  following  questions.  Give  the  completed  evaluation  to  your  industry 
representative  and  email  a  copy  to  NAVSUP  PERS-4412  Mr.  Phillip  Knauss  at  phillip.knauss@navy.mil.  Your  candid  feedback  will  be  key  to  improving  the 
program  for  future  participants. 


General 

1).  Give  example 
Comments: 

of  observed  innovative  business  approaches,  management  practices,  and  corporate  organizational  culture. 

2).  Give  example 
Navy. 

Comments: 

s  of  observed  senior  and  executive  level  decision  making  processess  that  are  different  from  the  Navy  and/or  may  be  beneficial  for  the 

3).  Give  example 
Comments: 

of  skills  developed  in  private  sector  precedures  and  best  practices  that  may  be  use  in  the  Navy. 

4).  Give  example 
Comments: 

of  how  you  contributed  to  the  strategic  and  service-level  linkage  with  public  policy,  corporate  and  non-government  organization. 

5).  Give  example 
Comments: 

of  how  your  ability  to  embrace  changes  and  other  cultures  in  a  complex  national  environment  has  enhanced. 

On  the  Job  Impact  and  Benefit 

6).  How  do  you  expect  to  use  what  you  learned  at  [FedEx]  in  the  Navy? 
Comments: 


7).  How  did  the  training  contribute  to  your  qualities  as  an  officer? 
Comments: 


8).  How  will  your  training  improve  your  quality  of  work  for  the  Navy? 
Comments: 


9).  What  aspect  of  TWI  was  least  valuable  or  not  likely  to  be  used? 
Comments: 


Personal  and  Professional  Value 

10).  What  area  of  the  FITREP  performance  trait  is  most  impacted  by  the  TWI  Program? 
o  33.  Professional  Expertise:  Professional  knowledge  proficiency  and  qualification 

o  34.  Command  or  Organizational  Climate/Equal  Opportunity:  Contributing  to  growth  and  development,  human  worth,  community 
o  35.  Military  Bearing  Character:  Appearance,  conduct  physical  fitness,  adherence  to  Navy  Core  Values 
o  36.  Teamwork:  Contributions  toward  team  building  and  team  results 

o  37.  Mission  Accomplishment  and  Initiative:  Taking  initiative,  planning/prioritizing,  achieving  mission 
o  38.  Leadership:  Organizing,  motivating  and  developing  others  to  accomplish  goals. 

Comments: 


11).  How  did  this  training  contribute  to  your  professional  goals? 
Comments: 


TWI  Program  Improvement 

Please  rank  from  1  (Entirely  ineffective)  to  5(Very  effective) 

12).  Distribution  of  time  on  different  content  and  skill  areas 

Comments: 

1  2  3  4  5 

13).  Length  of  the  program  and  the  balance  between  mentoring,  hands  on,  team  project,  etc 

Comments: 

1  2  3  4  5 

14).  Design  and  facilitation  of  learning  environment 

Comments: 

1  2  3  4  5 

15).  Connection  between  the  program  objectives  and  the  actual  learning  performance  1  2  3  4  5 

Comments: 


16).  What  should  be  added  to  the  program  to  enhance  the  learning?  Omitted? 
Comments: 
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Table  1 1 .  TWI  Program  Evaluation  Interpretation 


TWI  Program  Evaluation  Interpretation 

When  reviewingthe  answers  forthe  TWI  Program  Evaluation,  use  the  following  ranges  as  guidelines  for  responses  to  meeting  program 
expectations,  relevancy  of  training  to  NAVSUP  objectives,  and  overall  quality  of  program. 

General 

The  participant's  answer  to  the  "General"  section  demonstrated  if  TWI  program  have  filled  the  5  identified  capability  gaps. 

1)  Industry  innovative  business  approaches,  management  practices,  and  corporate  organizational  culture 

2)  Industry  senior  and  executive  level  decision-making  processes 

3)  Skills  in  private  sector  procedures  and  best  practices 

4)  Strategic  and  service-level  linkages  with  public  policy,  corporate  and  non-governmental  organizations 

5)  Ability  to  embrace  changes  and  other  cultures  in  a  complex  national  environment 

On  the  Job  Impact  and  Benefit 

The  participant's  answer  to  the  "On  the  Job  Impact  and  Benefit"  section  demonstrated  what  they  learned  and  how  they  plan  to  apply  it. 
This  will  capture  the  prospective  impact  to  the  Navy;  basically  the  benefit  of  TWI  program. 

Personal  and  Professional  Value 

The  participant's  answer  to  the  "Personal  and  Professional  Value"  section  demonstrated  if  the  participant  is  satisfied  with  TWI  program 
and  if  it  is  benefical  to  their  professional  career. 

TWI  Program  Improvement 

The  participant's  answer  to  the  "TWI  Program  Improvement"  section  will  help  sponsoring  company  improve  the  program  for  future 
participant.  This  will  also  help  the  Navy  draft  better  MOU/MOA  in  the  future. 


4.  Step  4:  TWI  Learning  and  Retention 

After  4-6  months,  TWI  participants  complete  another  post-program  evaluation 
(Table  12),  provided  by  PERS-4412.  This  evaluation  captures  whether  the  participant 
learned  and  retained  the  knowledge  learned  through  the  TWI  program.  The  4-6-month- 
period  of  settling  in  to  a  new  billet  and  re- acclimating  to  the  Navy  enables  the  participant 
to  reflect  on  how  the  TWI  experience  impacts  the  Navy.  The  participant  may  identify 
new  ways  in  which  he  is  productive,  efficient  and  a  better  officer  than  realized  after  the 
initial  completion  of  the  TWI  program.  Interpreting  the  4-6  month  post-program 
evaluation  can  be  found  in  Table  13. 
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Table  12.  TWI  Program  Evaluation  (4-6  Months) 


TWI  Program  Evaluation  (4-6  Months  ) 

Please  help  us  evaluate  the  Training  with  Industry  (TWI)  program  by  answering  the  following  questions.  Email  to  NAVSUP  PERS-4412  Mr.  Phillip  Knauss  at 
phillip.knauss(S)navy.mil.  Your  candid  feedback  will  be  key  to  improving  the  program  for  future  participants  and  capturing  the  benefit  of  TWI. 

General 

1).  Give  examples  of  observed  innovative  business  approaches,  management  practices,  and  corporate  organizational  culture. 

Comments: 


2) .  Give  examples  of  observed  senior  and  executive  level  decision  making  processess  that  are  different  from  the  Navy  and/or  may  be  beneficial  for  the 
Navy. 

Comments: 

3) .  Give  examples  of  skills  developed  in  private  sector  precedures  and  best  practices  that  may  be  use  in  the  Navy. 

Comments: 


4).  Give  examples  of  how  you  contributed  to  the  strategic  and  service-level  linkage  with  public  policy,  corporate  and  non-government  organization. 
Comments: 


5).  Give  examples  of  how  your  ability  to  embrace  changes  and  other  cultures  in  a  complex  national  environment  has  enhanced. 
Comments: 


On  the  Job  Impact  and  Benefit 

6).  How  do  you  expect  to  use  what  you  learned  at  [FedEx]  in  the  Navy?  What  knowledge  and  skill  from  TWI  have  you  used? 
Comments: 


7).  How  did  the  training  contribute  to  your  qualities  as  an  officer? 
Comments: 


8).  Did  the  training  improve  your  quality  of  work  forthe  Navy?  How? 
Comments: 


9).  What  aspect  of  TWI  was  least  valuable  or  not  used? 
Comments: 


Personal  and  Professional  Value 

10).  Was  area/s  of  the  FITREP  performance  trait  impacted  by  the  TWI  Program? 
o  33.  Professional  Expertise:  Professional  knowledge  proficiency  and  qualification 

o  34.  Command  or  Organizational  Climate/Equal  Opportunity:  Contributing  to  growth  and  development,  human  worth,  community 
o  35.  Military  Bearing  Character:  Appearance,  conduct  physical  fitness,  adherence  to  Navy  Core  Values 
o  36.  Teamwork:  Contributions  toward  team  building  and  team  results 

o  37.  Mission  Accomplishment  and  Initiative:  Taking  initiative,  planning/prioritizing,  achieving  mission 
o  38.  Leadership:  Organizing,  motivating  and  developing  others  to  accomplish  goals. 

Comments: 


11).  Did  the  training  contribute  to  your  expected  professional  goals? 
Comments: 


TWI  Program  Improvement 

Please  rank  from  1  (Entirely  ineffective)  to  5(Very  effective) 

12).  Distribution  of  time  on  different  content  and  skill  areas  1  2  3  4  5 

Comments: 


13).  Length  of  the  program  and  the  balance  between  mentoring,  hands  on,  team  project,  etc  1  2  3  4  5 

Comments: 


14).  Design  and  facilitation  of  learning  environment  1  2  3  4  5 

Comments: 


15).  Connection  between  the  program  objectives  and  the  actual  learning  performance  1  2  3  4  5 

Comments: 


16).  What  should  be  added  to  the  program  to  enhance  the  learning?  Omitted? 
Comments: 
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Table  13.  TWI  Program  Evaluation  Interpretation  (4-6  Months) 


TWI  Program  Evaluation  Interpretation  (4-6  Months) 

When  reviewing  the  answers  for  the  TWI  Program  Evaluation,  use  the  following  ranges  as  guidelines  for  responses  to  meeting  program 
expectations,  relevancy  of  training  to  NAVSUP  objectives,  and  overall  quality  of  program.  Giving  this  4-6  months  after  completion  of  TWI 
will  demonstrate  learning  and  retention  of  the  skills  and  knowledges  obtain  from  TWI. 

General 

The  participant's  answer  to  the  "General"  section  demonstrated  if  TWI  program  continues  to  fill  the  5  identified  capability  gaps. 

1)  Industry  innovative  business  approaches,  management  practices,  and  corporate  organizational  culture 

2)  Industry  senior  and  executive  level  decision-making  processes 

3)  Skills  in  private  sector  procedures  and  best  practices 

4)  Strategic  and  service-level  linkages  with  public  policy,  corporate  and  non-governmental  organizations 

5)  Ability  to  embrace  changes  and  other  cultures  in  a  complex  national  environment 

On  the  Job  Impact  and  Benefit 

The  participant's  answer  to  the  "On  the  Job  Impact  and  Benefit"  section  demonstrated  what  they  learned,  how  they  plan  to  apply  it,  and  if 
they  applied  it.  This  will  capture  the  prospective  impact  to  the  Navy;  basically  the  benefit  of  TWI  program. 

Personal  and  Professional  Value 

The  participant's  answertothe  "Personal  and  Professional  Value"  section  demonstrated  if  the  participant  is  satisfied,  unsatisfied,  or  still 
feels  the  same  with  TWI  program  and  if  it  is  benefical  to  their  professional  career. 

TWI  Program  Improvement 

The  participant's  answertothe  "TWI  Program  Improvement"  section  will  help  sponsoring  company  and  the  Navy  improve  the  program  for 
future  participant. 


5.  Step  5:  Navy  Impact 

Positive  impact  to  the  Navy  can  be  met  if  TWI  participants  are  sent  to  utilization 
tours  to  fulfill  billets  that  allow  them  an  opportunity  to  apply  learned  knowledge  and 
skills.  Hard  impact  can  be  captured  only  if  TWI  participants  lead  cost-saving  projects  or 
apply  new  concepts  to  improve  performance  or  procedure  efficiencies.  This  data  can  be 
captured  one  or  two  years  after  the  TWI  tour  by  using  Table  14.  The  benefit  captured  will 
be  applied  to  step  6  cost-benefit  analysis.  This  data  is  currently  not  available  for  this 
project. 
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Table  14.  Navy  Impacts 


STEP  5:  Navy  Impacts 

Cost  Saving  (dollar,  time,  etc) 

Cost  Saving  in  dollars 

Confidence  Level 

Benefit  Value 

1 

Example:  improved  warehouse 
restocking  time  by  50% 

Example:  Decrease  the 
time  by  50%  saved 
$50,000 

Example:  Used  the  method  learned  from  Home  Depot  while 
conducting  TWI  so  is  95%  confindence  this  saving  is  due  to 
knowledge  learned  at  TWI 

Example:  At  95%  confidence 
indicate  at  error  of  5%  so  the 
range  of  benefit  is  from  $47,500 
to  $52,500.  $47,500  is  use  for  a 

more  conservative  estimate. 

2 

3 

Total  Benefits  to  be  added  to  Step  6  CBA 

6.  Step  6:  Measure  All  Training  Costs  and  Benefits 

In  step  6,  all  costs  and  benefits,  to  include  intangible  benefits,  are  identified  and 
measured.  The  ROI  is  then  calculated  using: 

ROI  =  (Net  Program  Benefits/Program  Costs)  x  100 

Table  15  displays  the  tangible  and  intangible  benefits  identified  for  the  TWI 
program.  A  monetary  value  for  productivity  per  officer  is  5%  of  their  value  for  the  year. 
Monetary  values  for  fulfilling  identified  capability  gaps  is  20%  of  their  value  for  the  year. 
These  values  have  been  chosen  as  notional  values  and  are  thought  by  the  authors  and 
advisors  to  be  fair  representations  of  their  actual  values. 
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Table  15.  Navy  TWI  Program  Benefits 


Navy  TWI  Program  Benefits 

Benefits 

Tangible 

Quantifiable 

Tangible 

Non- 

Quantifiable 

Intangible 

Quantifiable 

Intangible 

Non- 

Quantifiable 

Monetary 

Value 

Productivity  Increase  per  Officer 

X 

$8,838.00 

Cost  Saving  Projects 

X 

$0.00 

Fulfilling  Identified  Capability  Gaps 
Per  Officer 

X 

$35,352.00 

Meeting  NAVSUP  Objectives 

X 

$0.00 

Professionally  a  better  officer  after  TWI 

tour 

X 

N/A 

Improved  civil-military  relation 

X 

N/A 

Increase  in  job  satisfaction 

X 

N/A 

Recruiting 

X 

N/A 

a.  Assumptions  Used  in  the  Analysis 

For  the  cost-benefit  analysis,  the  perspective  for  costs  and  benefits  is  that  of  the 
Navy.  All  pay  elements  are  based  on  the  FY  2017  Department  of  Defense  Military 
Personnel  Composite  Standard  Pay  and  Reimbursement  Rates,  using  the  Navy’s  annual 
DOD  Composite  Rate  (Roth,  2016).  The  requirement  for  TWI  selection  is  a  senior 
lieutenant  (0-3)  or  lieutenant  commander  (0-4).  Responses  from  a  TWI  participants’ 
survey  show  all  participants  as  0-4  with  at  least  10  years  of  service  except  for  one  0-2 
with  1-5  years  of  service,  who  was  an  outlier.  This  project  used  the  0-4  composite  rate 
for  the  cost  and  the  value  of  the  officer.  The  annual  DOD  composite  rate  includes  the 
“average  basic  pay  plus  retired  pay  accrual,  Medicare-eligible  retiree  health  care  accrual, 
basic  allowance  for  housing,  basic  allowance  for  subsistence,  incentive  and  special  pay, 
permanent  change  of  station  expenses,  and  miscellaneous  pay”  (Roth,  2016,  Note  2). 
Table  16  is  the  calculation  of  the  value  of  each  officer. 
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Table  16.  Officer’s  Value 


Value  by  Rank 

One  Officer 

Used  FY2017 # 

5%  skill 

increase 

20%  capability 
cost  saving 

Combined 

0-4 

$176,759.00 

$8,837.95 

$35,351.80 

$44,189.75 

0-5 

$198,950.00 

$9,947.50 

$39,790.00 

$49,737.50 

0-6 

$233,013.00 

$11,650.65 

$46,602.60 

$58,253.25 

Four  Officers 

0-4 

$176,759.00 

$35,351.80 

$141,407.20 

$176,759.00 

0-5 

$198,950.00 

$39,790.00 

$159,160.00 

$198,950.00 

0-6 

$233,013.00 

$46,602.60 

$186,410.40 

$233,013.00 

For  this  research,  an  assumption  was  made  of  the  TWI  participants  being  selected 
when  they  were  an  0-3  or  an  0-4  and  conducting  their  TWI  tour  during  their  13th  year  of 
service  (YOS).  It  is  also  assumed  that  they  would  promote  regularly  to  0-5  at  their  17th 
YOS  and  to  0-6  at  their  22nd  YOS.  Productivity  increase  after  a  TWI  tour  is  determined 
at  5%  based  on  a  study  conducted  by  Kamarck,  Thie,  Adelson,  and  Krull  (2010).  In  the 
study,  literature  review  shows  a  production  of  20-percent,  15-percent  from  domain 
knowledge  and  5-percent  from  skills  or  competencies  gained,  for  officers  who  attended  a 
one  and  a  half  year  of  graduate  school  program.  The  TWI  program  is  slightly  different 
from  a  graduate  study  with  more  focus  on  skill  so  this  project  used  5-percent  as  the 
productivity  increase  from  a  one-year  TWI  tour. 

According  to  the  survey  taken  by  TWI  participants,  many  gained  knowledge  and 
fulfilled  one  or  more  capability  gaps  identified  by  NAVSUP.  Assumption  is  that 
NAVSUP  would  have  to  hire  out  to  fulfill  the  capability  gaps.  By  TWI  participants 
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obtaining  the  knowledge  to  fulfil  the  capability  gaps,  there  is  an  assumption  of  cost 
saving  of  20-percent. 


b.  Calculation  of  Costs 

The  hard  cost  to  the  Navy  is  the  base  pay.  For  one  year,  the  Navy  is  paying  the 
four  selected  officers  their  pay  and  entitlements  to  work  in  corporate  America.  The  cost 
to  the  Navy  (0-4  rank)  is  $176,759  per  officer  per  year.  For  selection  of  four  officers  per 
year,  it  totals  $707,036.  Another  hard  cost  is  the  cost  of  the  selection  board,  which  is 
comprised  of  one  0-7,  four  0-6s,  two  0-5s,  one  0-4,  and  one  0-3.  These  officers 
conduct  read  ahead  of  candidates  and  meet  for  two  days  to  make  the  selection.  Using  the 
FY  2017  Navy  annual  DOD  composite  rate  and  multiplying  it  by  the  daily  rate  of 
0.00439  (Roth,  2017),  this  comes  out  to  the  following  cost  for  two  days  (excluding  travel 
and  per  diem  for  those  members  who  may  have  to  travel): 

One  0-7  =  $2,298 

Four  0-6s  =  $8,183 

Two  0-5s  =  $3,494 

One  0-4  =  $1,552 

One  0-3  =  $1,333 

Total  Selection  Board  Cost:  $16,860 

Because  most  cost  measurement  of  this  type  of  program  do  not  measure  the  cost 
of  the  selection  board,  this  calculation  is  available  but  will  not  be  included  in  the  ROI 
calculation.  This  will  ensure  an  easier  comparison  to  other  programs’  costs  and  benefits. 
This  brings  the  total  hard  cost  for  four  0-4s  for  the  Navy  to  $707,036. 

There  are  soft  costs  which  are  challenging  to  put  a  dollar  amount  on.  For  example, 
it  would  be  difficult  to  put  a  cost  on  the  loss  of  military  knowledge,  skills,  and  military 
bearing  of  the  officers  after  being  out  of  the  fleet  for  a  minimum  of  1 1  months.  Costs  that 
are  uncertain  and  unmeasurable  are  the  following:  (a)  the  cost  on  productivity  due  to  the 
officers  being  away  while  at  TWI  for  the  year;  (b)  quality  of  life  adversely  affected  by 

repeated  moves;  and  (c)  unnecessary  added  strain  and  stress  on  the  family  unit. 
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c.  Calculation  of  Benefits 

The  financial  benefit  to  the  Navy  is  the  increase  in  productivity  from  the  TWI 
participants  following  their  TWI  tour.  For  this  project,  5%  productivity  (equivalent  to 
$8,838  per  0-4  or  $35,352  for  four  0-4s)  is  applied  due  to  the  study  conducted  by  RAND 
(Kamarck  et  al.,  2010).  No  discount  rate,  time  value  of  money,  annual  pay  raise,  or 
inflation  is  taken  into  consideration  for  the  calculation. 

cl.  Intangible  Benefits 

In  addition  to  the  productivity  benefit,  there  are  intangible  benefits.  A  very 
important  measurable  benefit  is  the  fulfillment  of  the  identified  future  capability  gaps  and 
meeting  NAVSUP’s  four  goals  from  its  strategic  plan.  These  can  be  accurately  measured 
by  using  the  forms  provided  in  steps  2,  3,  and  4  of  the  Navy  TWI  ROI  model  with  future 
participants.  For  this  project,  only  the  fulfillment  of  the  capability  gaps  is  measured,  not 
the  meeting  of  NAVSUP’s  four  goals. 

Results  from  a  survey  given  in  2017  support  the  argument  that  most  of  these 
intangible  benefits  have  been  met.  Although  not  all  TWI  participants  will  meet  all  of  the 
identified  capability  gaps,  each  met  at  least  one  or  more  of  the  gaps  listed: 

•  Exposure,  training,  and  skills  in  industry  innovative  business  approaches, 
management  practices,  and  corporate  organizational  culture 

•  Exposure  to  industry  senior  and  executive  level  decision-making  processes 

•  Develop  skills  in  private  sector  procedures  and  best  practices 

•  Understand  strategic  and  service-level  linkages  with  public  policy, 
corporate  and  non-govemmental  organizations 

•  Ability  to  embrace  changes  and  other  cultures  in  a  complex  national 
environment.  (Stone,  2004;  Lyden,  2009;  Heinrich,  2012) 

Twenty  percent  of  the  officer’s  value  is  used  in  the  calculation.  For  example,  an 
0-4  capability  gaps  fulfillment  is  $35,352  per  officer  or  $141,407  per  four  officers.  This 
is  based  on  the  assumption  of  cost  saving  from  the  Navy  having  to  hire  and  train  to  fulfil 
the  existing  gaps. 
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One  hundred  percent  of  survey  replies  state  an  anticipated  commitment  of  at  least 
20  years  to  the  Navy.  Only  one  reply  (6%)  was  due  directly  to  the  participation  in  the 
TWI  program.  This  supports  the  argument  that  the  TWI  program  is  irrelevant  in  the 
increase  of  participants’  commitment  to  the  Navy,  but  since  participant  stayed  until  their 
20  years  obligation,  this  project  uses  the  benefit  at  20  years  of  service  (YOS)  for  the  ROI 
calculation  as  displayed  in  Table  17. 


Table  17.  Navy  TWI  ROI 


BENEFITS 

Year  of  Service  (YOS) 

13  YOS 

14  YOS 

15  YOS 

16  YOS 

17  YOS 

18  YOS 

19  YOS 

20  YOS 

21  YOS 

22  YOS 

23  YOS 

Investment 

TWI  Tour 

Payback 

Yearl 

Payback 
Year  2 

Payback 
Year  3 

Reg  tour 

Promote  to  05 

Reg  tour 

Reg  tour 

Reg  tour 

Reg  tour 

Reg  tour 

Promote  to  06 

Reg  tour 

Benefits:  One  0-4  officer 

Skill  productivity  of  5% 

($176,759.00) 

($167,921.05) 

($132,569.25) 

($123,731.30) 

($113,783.80) 

($103,836.30) 

($93,888.80) 

($83,941.30) 

($73,993.80) 

($62,343.15) 

($50,692.50) 

Cost  saving  from  filling  capability  gap  at 
20% 

($176,759.00) 

($141,407.20) 

($106,055.40) 

($70,703.60) 

($30,913.60) 

$8,876.40 

$48,666.40 

$88,456.40 

$128,246.40 

$174,849.00 

$221,451.60 

Total  benefits  (Productivity  and 
capability  gaps) 

($176,759.00) 

($132,569.25) 

($88,379.50) 

($44,189.75) 

$5,547.75 

$55,285.25 

$105,022.75 

$154,760.25 

$204,497.75 

$262,751.00 

$321,004.25 

Benefits:  Four  0-4  officer 

Skill  productivity  of  5% 

($707,036.00) 

($671,684.20) 

($636,332.40) 

($600,980.60) 

($561,190.60) 

($521,400.60) 

($481,610.60) 

($441,820.60) 

($402,030.60) 

($355,428.00) 

($308,825.40) 

Cost  saving  from  filling  capability  gap  at 
20% 

($707,036.00) 

($565,628.80) 

($424,221.60) 

($282,814.40) 

($123,654.40) 

$35,505.60 

$194,665.60 

$353,825.60 

$512,985.60 

$699,396.00 

$885,806.40 

Total  benefits  (Productivity  and 
capability  gaps) 

($707,036.00) 

($530,277.00) 

($353,518.00) 

($176,759.00) 

$22,191.00 

$221,141.00 

$420,091.00 

$619,041.00 

$817,991.00 

$1,051,004.00 

$1,284,017.00 

***  Add  benefit  realized  from  Step  5  into  the  year  it  is  realized.  *** 

ROI  =  (Net  Program  Benefits  /  Program  Costs)  x  100— >  (619,041  /  707,036)  *  100 


The  net  program  benefit  at  20  YOS  is  then  applied  to  the  formula  from  Phillips 
(1997)  to  determine  an  ROI  percentage. 

ROI  =  (Net  Program  Benefits/Program  Costs)  x  100 

ROI  =  (NPV/Program  Costs)  x  100 

ROI=  (619,041  /  707,036)  x  100  =  87.5544, 

which  means  for  each  dollar  invested  in  the  TWI  program,  the  Navy  received  $0.88 
dollars  in  return  after  the  cost  of  the  program  had  been  recouped.  In  the  long  term,  Navy 
TWI  is  a  benefit  to  the  Navy.  Three  years  after  conducting  the  TWI  tour  (17  YOS),  the 
investment  would  be  recovered.  See  Table  17. 
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Other  intangible  benefits  that  are  non-quantifiable  are  (a)  the  human  capital 
investment,  which  is  the  theoretical  better  leader,  more  efficient  officer,  more  business 
savvy  officer  returning  to  the  fleet;  (b)  the  improvement  of  the  civil-military  relation. 
According  to  Thompson  (2011),  there  is  larger  separation  between  civilian  and  military 
leadership  which  may  be  problematic  to  America’s  democracy.  The  TWI  program 
contributes  to  the  improvement  of  the  civil-military  relations  through  the  one  year 
interaction;  (c)  increase  in  job  satisfaction  and  commitment  to  the  Navy;  and  (d)  a  tool 
for  recruiting  future  leaders  to  the  Navy.  Results  from  the  survey  conducted  with 
participants  in  2017  shows  a  positive  human  capital  investment  benefit.  Seventy-three 
percent  of  participants  saw  immediate  value  added  and  60-percent  found  interaction  with 
TWI  executives  to  be  a  great  extent  of  value  added  to  their  military  career. 

e.  Sensitivity  Analysis 

In  the  ROI  analysis,  it  assumed  that  TWI  tour  provided  5%  productivity  per 
officer  based  on  similar  study  conducted  by  Kamarck  et  al.  (2010).  It  was  also  assumed 
that  the  cost  saving  from  having  to  hire  and  train  to  fulfill  identified  capability  gaps  is 
20%.  There  are  many  factors  that  can  affect  the  percentage.  A  sensitivity  analysis  was 
conducted  with  varying  rate.  A  low  of  4%  skill  increase  and  15%  capability  cost  saving,  a 
medium  of  5%/20%  and  a  high  of  6%/25%  was  examined.  Table  18  displays  the  value 
for  each  rank. 


Table  18.  Sensitivity  Analysis  Value  by  Rank 


Value  by  Rank 

Four  Officers 

Used  FY2017  # 

4%  skill 

increase 

15%  capability 
cost  saving 

Combined 

5%  skill 

increase 

20%  capability 
cost  saving 

Combined 

6%  skill 

increase 

25%  capability 
cost  saving 

Combined 

0-4 

$176,759.00 

$28,281.44 

$106,055.40 

$134,336.84 

$35,351.80 

$141,407.20 

$176,759.00 

$42,422.16 

$176,759.00 

$219,181.16 

0-5 

$198,950.00 

$31,832.00 

$119,370.00 

$151,202.00 

$39,790.00 

$159,160.00 

$198,950.00 

$47,748.00 

$198,950.00 

$246,698.00 

0-6 

$233,013.00 

$37,282.08 

$139,807.80 

$177,089.88 

$46,602.60 

$186,410.40 

$233,013.00 

$55,923.12 

$233,013.00 

$288,936.12 
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Table  19  shows  the  calculation  for  the  low,  medium,  and  high  rate.  Figure  9 
displays  the  low,  medium,  and  high  for  one  officer’s  rate  of  return.  All  rates  obtained 
ROI  at  20  YOS.  The  three-year  utilization  tour  is  sufficient  to  pay  back  the  investment, 
but  participant  will  need  to  conduct  an  additional  tour  for  the  Navy  to  start  reaping 
benefit. 


Table  19.  Sensitivity  Analysis  Benefit  Calculations 


BENEFITS 

Year  of  Service  (YOS) 

13  YOS 

14  YOS 

15  YOS 

16  YOS 

17  YOS 

18  YOS 

19  YOS 

20  YOS 

21  YOS 

22  YOS 

23  YOS 

Four  Officers  per  year 

Investment 

TWI  Tour 

Payback 

Yearl 

Payback 

Year  2 

Payback 

Year  3 

Reg  tour 

Promote  to  05 

Reg  tour 

Reg  tour 

Reg  tour 

Reg  tour 

Reg  tour 

Promote  to  06 

Reg  tour 

LOW 

Skill  productivity  of  4% 

($707,036.00) 

($678,754.56) 

($650,473.12) 

($622,191.68) 

($590,359.68) 

($558,527.68) 

($526,695.68) 

($494,863.68) 

($463,031.68) 

($425,749.60) 

($388,467.52) 

Cost  saving  from  filling  capability  gap  at 
15% 

($707,036.00) 

($600,980.60) 

($494,925.20) 

($388,869.80) 

($269,499.80) 

($150,129.80) 

($30,759.80) 

$88,610.20 

$207,980.20 

$347,788.00 

$487,595.80 

Total  benefits  (Productivity  and 
capability  gaps) 

($707,036.00) 

($572,699.16) 

($438,362.32) 

($304,025.48) 

($152,823.48) 

($1,621.48) 

$149,580.52 

$300,782.52 

$451,984.52 

$629,074.40 

$806,164.28 

MEDIUM 

Skill  productivity  of  5% 

($707,036.00) 

($671,684.20) 

($636,332.40) 

($600,980.60) 

($561,190.60) 

($521,400.60) 

($481,610.60) 

($441,820.60) 

($402,030.60) 

($355,428.00) 

($308,825.40) 

Cost  saving  from  filling  capability  gap  at 
20% 

($707,036.00) 

($565,628.80) 

($424,221.60) 

($282,814.40) 

($123,654.40) 

$35,505.60 

$194,665.60 

$353,825.60 

$512,985.60 

$699,396.00 

$885,806.40 

Total  benefits  (Productivity  and 
capability  gaps) 

($707,036.00) 

($530,277.00) 

($353,518.00) 

($176,759.00) 

$22,191.00 

$221,141.00 

$420,091.00 

$619,041.00 

$817,991.00 

$1,051,004.00 

$1,284,017.00 

HIGH 

Skill  productivity  of  6% 

($707,036.00) 

($664,613.84) 

($622,191.68) 

($579,769.52) 

($532,021.52) 

($484,273.52) 

($436,525.52) 

($388,777.52) 

($341,029.52) 

($285,106.40) 

($229,183.28) 

Cost  saving  from  filling  capability  gap  at 

25% 

($707,036.00) 

($530,277.00) 

($353,518.00) 

($176,759.00) 

$22,191.00 

$221,141.00 

$420,091.00 

$619,041.00 

$817,991.00 

$1,051,004.00 

$1,284,017.00 

Total  benefits  (Productivity  and 
capability  gaps) 

($707,036.00) 

($487,854.84) 

($268,673.68) 

($49,492.52) 

$197,205.48 

$443,903.48 

$690,601.48 

$937,299.48 

$1,183,997.48 

$1,472,933.60 

$1,761,869.72 

***  Add  benefit  realized  from  Step  5  into  the  year  it  is  realized.  *** 

ROI  =  (Net  Program  Benefits  /  Program  Costs)  x  100 

LOW:  43%  MED:  88%  High:  133% 
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Figure  9.  Sensitivity  Analysis  for  One  Officer 

7.  Step  7:  Full  Training  Report 

As  previously  stated  in  Chapter  II,  bringing  all  the  information  together  will 
provide  a  comprehensive  view  of  the  ROI  the  Navy  gets  from  the  TWI  program.  To 
communicate  everything  important  about  the  investment,  all  material  from  steps  1-6 
must  be  presented.  Identifying  objectives,  creating  a  scorecard  linking  objectives, 
determining  how  participants  apply  what  was  learned,  placing  a  value  on  meeting 
objectives,  and  determining  the  cost-benefit  analysis  will  together  provide  the  overall 
ROI  for  the  TWI  program  (Schmidt,  2016).  This  project  was  able  to  capture  the  hard 
ROI,  but  not  the  soft  ROI  so  the  overall  ROI  for  TWI  was  not  captured.  Using  provided 
forms  to  collect  data  with  future  participants  will  provide  a  more  accurate  calculation  for 
costs  and  benefits 


Sensitivity  Analysis: 

RATE  OF  RETURN  FOR  ONE  OFFICER 


PRODUCTIVITY 

low 

Medium 
- high 

BREAKEVEN 

18  19  20 


YEARS  OF  SERVICE 
TWI  THROUGH  20  YEARS 

YEAR  13  =  INITIAL  INVESTMENT  IN  THE  TWI  PROGRAM 
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B.  CHAPTER  SUMMARY 


Simply  going  straight  to  the  ROI  formula  will  not  be  sufficient  for  the 
measurement  of  a  leadership  training  skill.  This  MBA  project  utilizes  the  Navy  TWI  ROI 
Model,  which  is  a  combination  of  the  three  methodologies  discussed  in  Chapter  II.  The 
ROI  is  88%.  Using  the  survey  conducted,  it  can  be  determined  that  the  intangible  benefit 
of  fulfilling  the  identified  capability  gaps  were  met,  the  increased  commitment  to  the 
Navy  due  to  TWI  was  not  evident,  and  the  human  capital  investment  was  met.  For  a  more 
thorough  and  accurate  measurement  of  the  intangible  benefits  in  the  future,  it  would  be 
beneficial  for  the  Navy  to  use  the  forms  provided  in  Step  2  (Scorecard),  Step  3  (Initial 
TWI  Program  Evaluation),  and  Step  4  (4-6  Months  TWI  Program  Evaluation).  Overall, 
the  TWI  program  is  determined  to  be  beneficial  for  the  Navy. 
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V.  SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 

A.  SUMMARY 

Applying  the  right  methodology  to  the  right  training  will  enable  decision-makers, 
training  staff,  comptrollers,  and  executives  to  determine  whether  training  returns  are 
worth  the  investment.  In  attempting  to  develop  an  ROI  metric  for  NAVSUP  to  apply  to 
its  TWI  program,  this  research  established  three  ROI  methodologies.  Through  the 
combination  of  elements  of  the  three  methodologies,  a  metric  was  developed  that  would 
best  address  the  ROI  question  for  the  Navy  Supply  Corps.  The  metric  is  adaptable  to  the 
changing  needs  of  the  Navy,  allowing  users  to  add  and  take  away  objectives  as  necessary. 

The  research  designed  and  sent  a  survey  to  the  participants  in  the  Navy  Supply 
Corps  TWI  program.  The  spirit  of  the  survey  was  to  gather  reactions  to  training, 
utilization  of  training,  and  impact  to  career  milestones.  The  survey  was  able  to  solicit 
response  from  15  of  24  TWI  participants. 

Research  questions  were  addressed  using  the  survey,  Navy  policy  guidance, 
career  planning  milestones  and  promotion  opportunity  as  identified  through  the  project. 

B.  CONCLUSIONS  AND  RECOMMENDATIONS 

1.  What  Are  the  Costs  and  Benefits  of  the  TWI  Program  to  the  Navy? 

a.  Conclusion 

The  cost  of  the  TWI  program  to  the  Navy  is  the  salary  and  benefits  paid  to  one 
officer  while  at  the  one  year  TWI  tour.  Therefore,  the  cost  to  send  one  0-4  through  the 
TWI  program  is  $176,759.  Determining  the  benefit  to  the  Navy  as  a  5%  skill  increase 
($8,838)  and  a  20%  cost  saving  in  filling  capability  gaps  ($35,352),  the  yearly  benefit  of 
sending  an  0-4  through  the  TWI  program  is  $44,190.  Over  a  20  year  career  and  factoring 
promotion  within  zone,  applying  the  ROI  formula  ROI  =  (Net  Program  Benefits/Program 
Costs)  x  100,  the  ROI  for  the  TWI  program  was  determined  to  be  88%  (Table  19). 
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b.  Recommendations 

NAVSUP  should  use  the  Navy  TWI  ROI  Model  (Appendix  C)  to  holistically 
evaluate  the  TWI  program.  A  cost/benefit  analysis  does  not  take  into  account  the  entire 
program.  Using  forms  like  the  TWI  Scorecard  (Appendix  E),  TWI  Program  Evaluations 
(Appendices  E  and  F),  and  compiling  all  the  data  into  the  TWI  Full  Report  (Appendix  I) 
will  improve  the  criteria  to  evaluate  the  overall  benefit. 

2.  What  Is  the  Retention  Rate  for  TWI  Fellows  after  the  Three- Year 

Utilization  Tour? 

a.  Conclusion 

The  Navy  Supply  Corps  TWI  eligibility  requirements  (NAVSUP,  2013)  and 
selection  process  of  TWI  participants  is  effective  in  selecting  officers  committed  to  a  full 
20-year  or  more  career.  All  survey  respondents  indicated  that  they  intend  to  stay  in  the 
Navy  for  at  least  20  years.  Therefore,  based  on  the  survey,  an  assumption  is  that  upon 
completion  of  a  TWI  utilization  tour,  retention  of  participants  is  100%. 

b.  Recommendations 

NAVSUP  should  require  the  three-year  utilization  tour  after  the  TWI  program  be 
served  separate  from  other  payback  tours.  An  officer  completing  a  Navy-funded 
postgraduate  education  program  typically  owes  a  three-year  payback  tour.  Seventy-three 
percent  of  the  survey  respondents  owed  a  utilization/payback  for  both  TWI  and  graduate 
school.  These  payback  tours  are  served  concurrently.  Separating  the  payback  tours  would 
commit  an  officer  to  six  years  of  additional  service  versus  the  present  three  concurrent 
years. 


3.  Does  the  Utilization  Tour  Fill  the  Correct  Billet  and  Meet  the 
Professional  Requirement? 

a.  Conclusion 

The  utilization  tour  does  not  fill  the  correct  billet  or  meet  the  professional 


requirement.  While  the  Supply  Corps  has  suggested  certain  types  of  billets  as  utilization 
tours,  there  are  no  specific  billets  identified.  Current  utilization  tours  have  NOBC  codes 
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ranging  from  equipment  program  support  officer  (1920),  inventory  control  methods 
officer  (1515),  procurement  contracting  officer  (1480),  to  supply  plans  officer  (1984)  (P. 
Knauss,  personal  communication,  March  6,  2017).  Suggested  locations  for  utilization  are 
identified.  Many  report  to  a  NAVSUP  Weapons  Systems  Center  for  utilization;  however, 
Defense  Logistics  Agency  (DLA),  United  States  Transportation  Command 
(TRANSCOM),  and  U.S.  Fleet  Forces  Command  are  just  a  few  of  the  other  potential 
utilization  sites. 

b.  Recommendations 

1.  NAVSUP  should  assign  a  specific  NOBC  to  TWI  billets  instead  of  the 
generic  student  code  of  3289.  Each  TWI  billet  should  have  a  code(s) 
identifying  the  skills,  training,  experience,  and  capabilities  expected  to  be 
gained  from  the  experience.  TWI  participants  should  then  go  to  a 
utilization  tour  with  at  least  one  matching  NOBC  code  they  obtained  from 
the  TWI  program. 

2.  NAVSUP  should  establish  a  new  AQD  for  those  who  have  completed  the 
TWI  program.  NAVSUP  should  also  have  specific  billets  identified  and 
coded  for  the  new  AQD.  The  generic  24G  AQD  currently  assigned  after 
TWI  completion  does  not  have  any  specific  Supply  Corps  billet  requiring 
the  AQD.  See  Appendix  B  to  establish  an  AQD. 

3.  NAVSUP  should  require  TWI  participants  going  to  Starbucks  or  FedEx 
possess  the  1302  subspecialty  for  supply  chain  management,  which  aligns 
with  the  ExxonMobil  prerequisite  of  participants  requiring  the  Petroleum 
Management  1307  subspecialty  code. 

4.  NAVSUP  should  designate  a  NAVSUP  command  as  a  utilization  tour  for 
the  TWI  program.  This  would  provide  participants  with  more  exposure  to 
top-level  Supply  Corps  leadership.  The  leadership  can  in  turn  see  the 
benefits  to  the  TWI  program  and  offer  areas  for  improvement  as  the  needs 
of  the  Supply  Corps  change. 

4.  Do  the  Skills/Competencies  Obtained  through  TWI  Meet  the  DOD’s 
Guidance? 

a.  Conclusion 

The  Navy  Supply  Corps  TWI  program  meets  most  of  the  guidance  laid  out  by  the 
DOD  in  its  instruction  1322.06,  Fellowships,  Scholarships,  Training  with  Industry  (TWI), 
and  Grants  for  DOD  Personnel  (Table  20). 
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Table  20.  Navy  Supply  Corps  Fulfilling  DOD  TWI  Criteria.  Adapted  from 

Department  of  Defense  (2007). 


Department  of  Defense  Instruction  1322.06  (DOD,  2007,  p. 

8) ,  military  personnel  may  be  accepted  into  TWI  only 
under  the  following  circumstances: 

YES 

NO 

In  recognition  of  outstanding  performance  in  their  fields 

X 

To  undertake  a  project  that  may  be  of  value  to  the  United 

States 

X 

For  development  of  their  recognized  potential  for  future  career 
service 

X 

To  acquire  a  skill,  knowledge,  or  ability  to  fulfill  a  present 
need,  anticipated  requirement,  or  future  capability  that 
contributes  to  the  effectiveness  of  the  respective  Military 
Department  and  contributes  to  the  transformation  of  the 
Department  of  Defense 

X 

The  criteria  for  an  assignment  to  the  TWI  program  include 
the  following  (pp.  9-10): 

YES 

NO 

There  must  be  an  existing  Military  Component  need  or  desired 
future  capability  fulfilled  by  virtue  of  the  experience  gained 

X 

There  must  be  either  a  follow-on  utilization  tour  or  assignment 
to  which  the  individual  shall  be  assigned,  or  a  clear, 
documented  future  need  for  the  skill  that  is  gained. 

X 

If  individuals  completing  the  TWI  are  not  immediately  placed 
in  a  utilization  tour  or  assignment,  the  Secretary  concerned 
shall  keep  administrative  oversight  of  the  individuals  and  their 
gained  skills  for  utilization  at  a  time  determined  by  the 

Military  Component 

X 

The  TWI  tour  or  assignment  should  not  exceed  12  months  in 
length.  Any  TWI  tour  or  assignment  that  will  exceeds  12 
months  must  be  approved  by  the  Secretary  concerned. 

X 

The  proposed  TWI  tour  or  assignment  must  meet  professional 
development  requirements 

X 

There  must  be  a  written  agreement  between  the  private  sector 
host,  the  employee,  and  the  DOD  Component  concerned 
before  the  start  of  the  TWI  assignment 

X 

The  education  or  training  to  be  received  or  the  research  to  be 
performed  by  a  fellowship,  scholarship,  TWI,  or  grant 
recipient  must  be  designed  to  qualify  the  recipient  to  satisfy  a 
requirement  or  potential  requirement  of  the  Department  of 
Defense,  contribute  to  the  recipient’s  recognized  potential  for 
career  service,  or  constitute  a  contribution  to  a  project  of  value 
to  the  United  States 

X 
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b.  Recommendations 

1.  The  gaining  command  for  a  TWI  participants  should  assign  the  officer  to  a 
project  upon  completion  of  the  TWI  program.  According  to  Bennington 
and  Laffoley  (2012),  a  helpful  way  to  assess  the  effectiveness  of  a  training 
program  is  to  assign  participants  to  an  actual  project.  Identifying  specific 
TWI  program  outcomes,  such  as  leadership/management  skills, 
development  of  effective  strategies,  or  development  of  communication 
skills  can  all  be  demonstrated  and  measured  in  the  isolation  of  a  specific 
project. 

2.  NAVSUP  should  send  TWI  selectees  to  their  utilization  tour  for  one  year 
prior  to  the  TWI  program.  After  the  TWI  program,  they  return  to  the 
utilization  tour  for  the  remaining  two  years.  Working  for  the  utilization 
command  prior  to  the  TWI  program  enables  continuity  and  the  ability  to 
see  effects  of  the  training  on  the  individual  participant. 


5.  What  Are  the  Effects  of  the  TWI  Program  on  the  Officer’s  Promotion 
and  Career? 

a.  Conclusion 

There  are  no  major  effects,  positive  or  negative,  on  an  officer’s  promotion  or 
career  after  the  TWI  program.  Based  on  the  consent  of  12  TWI  participants,  data  on 
promotion  history  was  obtained  from  Naval  Personnel  Command.  These  participants  are 
promoting  within  their  “in-zone”  period. 

b.  Recommendation 

There  is  no  recommendation  for  this  conclusion. 

C.  FURTHER  RESEARCH 

The  scope  and  time  for  this  project  was  constrained.  Three  ROI  methodologies 
were  thoroughly  researched,  but  only  a  survey  was  used  to  gather  data  from  past 
participants  in  the  TWI  program.  Further  research  is  recommended  using  forms 
developed  from  this  project  to  thoroughly  capture  true  costs  and  benefits.  Further 
improvement  can  be  applied  to  the  ROI  process  developed  in  this  project. 
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A  second  area  for  further  research  in  the  benefit  of  assigning  specific  utilization 
tours  and  AQDs.  The  following  are  some  questions  to  explore:  (a)  Are  more  benefits 
gained  from  sending  a  participant  to  a  specific  billet?;  (b)  are  participants  more 
productive  at  one  billet  than  another?  and  (c)  is  TWI  more  attractive  to  an  officer  due  to 
gaining  an  AQD? 

Finally,  with  the  new  retirement  system  going  into  effect  January  2018,  retention 
of  TWI  participants  should  be  studied  as  there  may  be  less  incentive  for  officers  to  stay  in 
the  Navy  for  at  least  20  years. 
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APPENDIX  A.  ESTABLISHING  OR  REVISING  SUBSYSTEMS 

(NOBC)  WITHIN  NOOCS 


The  NOOCS  Board  is  the  central  location  for  changes  to  the  NOOCS.  To  propose  the 
establishment,  deletion,  or  revision  of  an  NOOCS  subsystem,  send  recommendations  via 
the  chain  of  command  to: 

Commanding  Officer  Navy  Manpower  Analysis  Center  (Attn:  Code  10) 

5722  Integrity  Drive 
Millington,  TN  38054-5011 

Recommendations  may  also  be  emailed  to  navmac  noocs@navy.mil 

Navy  Officer  Billet  Classification  (NOBC) 

To  establish  an  NOBC,  the  Manual  of  Navy  Officer  Manpower  and  Personnel 
Classifications  Volume  I  states  to  include: 

1.  Recommended  code  number,  long  and  short  title.  The  maximum  length  for  the 
short  title  is  14  spaces,  including  blank  spaces. 

2.  Recommended  definition  (preferably  limited  to  fifty  words); 

3.  Number  and  location  of  billets  by  activity  type  that  the  proposed  NOBC  will 
identify; 

4.  Designators  and  grades  applicable  to  the  proposed  code; 

5.  List  of  special  training  required  for  the  incumbent  officer  (include  length  and 
type  of  course(s)  and/or  on-the-job  training); 

6.  Information  on  how  the  requirements/qualifications  are  currently  identified; 

7.  Justification  stating  why  the  existing  billet  descriptors  are  insufficient  without 
the  proposed  NOBC,  and  other  historical  or  amplifying  information  necessary 
for  prudent  consideration  at  each  level  of  review;  and 

8.  Point  of  contact  (include  telephone  (both  commercial  and  DSN  number,  fax 
number,  and  e-mail  address).  (DON,  2017a,  p.  C-5) 

To  revise  an  NOBC,  the  Manual  of  Navy  Officer  Manpower  and  Personnel 
Classifications  Volume  I  states  to  include: 

1.  Code  and  title  of  existing  NOBC; 

2.  Proposed  recoding,  retitling,  and/or  rewording  of  the  current  NOBC; 

3.  Justification  for  the  proposed  revision(s);  and 

4.  Point  of  contact.  (DON,  2017a,  p.  C-5) 
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APPENDIX  B.  ESTABLISHING  OR  REVISING  SUBSYSTEMS 

(AQD)  WITHIN  NOOCS 


The  NOOCS  Board  is  the  central  location  for  changes  to  the  NOOCS.  To  propose  the 
establishment,  deletion,  or  revision  of  an  NOOCS  subsystem,  send  recommendations  via 
the  chain  of  command  to: 

Commanding  Officer  Navy  Manpower  Analysis  Center  (Attn:  Code  10) 

5722  Integrity  Drive 
Millington,  TN  38054-501 1 

Recommendations  may  also  be  emailed  to  navmac  noocs@navy.mil 

Additional  Qualification  Designation  (AQD) 

To  establish  an  AQD,  the  Manual  of  Navy  Officer  Manpower  and  Personnel 
Classifications  Volume  1  states  the  following  criteria  must  be  met: 

1.  It  does  not  duplicate  an  existing  code. 

2.  It  has  practical  application  in  personnel  planning,  control,  career  management, 
and  training. 

3.  It  identifies  additional  qualifications,  skills,  and  knowledge  required  by  the 
duties  and  functions  of  the  billet,  peacetime  and  mobilization,  and/or 
additional  knowledge,  skills,  and  qualifications  attained  by  officers,  both 
active  and  inactive. 

4.  It  must  identify  qualifications,  skills,  and  knowledge  that  cannot  be  identified 
by  a  designator,  grade,  NOBC,  or  subspecialty  code. 

5.  It  identifies  a  minimum  of  15  billets  or  15  officers.  (DON,  2017a,  pp.  D-3) 

Guidance  for  submitting  a  proposal  to  establish  an  AQD  code  is  found  in  the  Manual  of 
Navy  Officer  Manpower  and  Personnel  Classifications  Volume  I  and  states; 

a.  Commands  recommending  the  establishment,  deletion,  or  revision  of  AQD  codes  must 
ensure  proposals  meet  the  AQD  criteria  [in  the  prior  list]. 

(1)  Proposals  to  establish  or  revise  AQD  codes  must  be  submitted  as  endorsed 
letters  and  include  the  following  information.  Proposals  to  revise  AQD(s)  should  include 
all  the  items  below  but  may  state  “no  change”  for  those  items  that  are  the  same  as  the 
existing  AQD(s). 

(a)  Recommended  code  number. 

(b)  Titles  for  second  and  third  AQD  characters. 

(c)  Billet  detailing  requirements.  Indicate  whether  or  not  billets  have  been 
coded  to  require  an  officer  who  has  earned  the  AQD.  The  description 
should  include  special  notes  as  appropriate. 
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(d)  Officer  awarding  criteria.  Specify  whether  the  AQD  is  earned  by 
course  completion,  certification,  successful  completion  of  a  tour  in  a 
qualifying  billet,  or  a  combination  thereof.  If  a  tour  must  be  completed, 
identify  minimum  length  of  tour. 

(e)  Length  of  validity.  Specify  whether  or  not  the  AQD  expires  or  is 
indefinite.  If  it  expires,  include  (1)  the  time  period  for  which  the  AQD  is 
valid  and  (2)  any  recertification  requirements. 

(f)  Designators  that  can  hold  the  AQD. 

(g)  Minimum  and  maximum  Grades. 

(h)  Statement  as  to  whether  the  AQD  is  for  Active  Only,  Reserve  Only,  or 
Active  and  Reserve. 

(i)  Primary  consultant(s). 

(j)  Auxiliary  consultant(s). 

(k)  Funding  implications.  Resource  Sponsor  endorsement  must 
accompany  all  proposals  that  have  lasting  funding  implications.  ALL 
funding  implications  MUST  be  clearly  stated  as 
PROGRAMMED/APPROVED. 

(l)  Cost  Analysis.  What  will  be  the  net  effect  on  students  [Individuals 
Account]  IA  in  terms  of  man-years/Manpower  Personnel  Navy  (MPN)? 
NOTE:  MPN  man-years  in  the  student  IA  are  calculated  by  multiplying 
course  length  in  days  (include  weekends)  by  number  of  students  and 
dividing  that  by  365  for  initial  training  in  man-years.  This  figure  is  divided 
by  3  for  approximate  follow-on,  per-year  cost  for  out  years. 

Course  length  in  days  *  number  of  billets  MPN 

MPN  = - - - - - Approx  follow-on  = - 

365  3 

(m)  Number  and  location  of  billets  by  activity  type  that  will  be  identified 
by  the  proposed  AQD. 

(n)  Information  on  how  the  qualifications  are  presently  identified. 

(o)  Justification  as  to  why  existing  billet  descriptors  (i.e.,  NOBC  or  SSP 
codes)  are  insufficient. 

(p)  Point  of  contact  (include  telephone  (commercial  and  DSN)  numbers 
and  e-mail  address). 

(2)  A  code  table  must  be  enclosed  with  each  proposal  and  should  be  prepared  per 
the  template  at  the  end  of  this  section. 

NOTE:  Part  D  of  the  manual  is  currently  undergoing  a  complete  revision  in  terms 
of  content  and  format,  specifically  regarding  the  code  tables.  Please  make  note  of 
the  new  template  for  establishment  and  revision  requests.  Additionally,  due  to 
existing  AQD  code  tables  in  their  previous  formats,  there  are  cases  where  the 
AQD  order  does  not  follow  logical  alpha-numeric  sequencing.  Please  ensure  you 
view  the  entire  section  for  completeness  when  searching  for  a  specific  AQD  as  it 
may  be  out  of  sequence  until  the  entire  manual  can  be  formatted  appropriately. 


74 


(3)  Proposals  to  disestablish  AQD  codes  must  be  submitted  as  endorsed  letters 
and  include  the  following  information:  (1)  reason(s)  for  deletion  (e.g.,  qualifications 
duplicated  by  existing  designator,  sufficiently  identified  by  means  other  than  AQD,  or 
insufficient  requirements  reflected  on  manpower  authorizations)  and  (2)  other 
information  necessary  for  prudent  consideration  at  each  level  of  review. 

(4)  Due  to  data  integrity  over  time  and  the  risk  of  future  personnel 
misrepresentation,  best  practices  mandate  that  classification  elements  are  not  to  be  reused 
once  they  are  disestablished.  To  this  end,  deleted  or  disestablished  AQDs  will  not  be 
reused  at  any  time  unless  they  are  reestablished  under  the  same  criteria  and  guidelines  as 
previously  published. 

b.  Submit  proposals  to  the  address  listed  [at  the  beginning  of  Appendix  B].  After 
approval  of  a  proposal,  the  originator  must: 

(1)  Coordinate  with  Budget  Submitting  Office(s)  (BSOs)  to  ensure  Total  Force 
Manpower  Management  System  (TFMMS)  package(s)  are  submitted  with  Billet  Change 
Requests  (BCRs)  to  implement  changes. 

(2)  Liaise  with  NAVPERSCOM  (PERS-4)  to  ensure  officers  receive  codes  for 
assignment  completed  prior  to  code  establishment.  (DON,  2017a,  pp.  D-3  -  D-4) 


TEMPLATE  FOR  AQD  CODE  TABLE 


1ST 


CHARACTER 

2ND 


OFFICER  AWARDING 
CRITERIA 


BILLET  DETAILING 
PREREQUISITES 


CONSULTANTS 


CODE 


TITLE 


CODE 


TITLE 


Officer  will  be  awarded  this 
AQD  if  member  has: 


If  other  than  “N/A",  then  there  are  existing  billet 
requirements  coded  in  TFMMS. 


How  is  the  AQD  awarded?  Explain 
whether  the  AQD  is  awarded  by  course 
completion,  certification,  successful 
completion  of  a  tour  in  a  qualifying 
billet,  or  a  combination  thereof.  If  a  tour 
must  be  completed,  identify-  minimum 
tour  length.  If  available  to  Reserve 
Component  Officers,  specify-  Reserve- 
specific  awarding  criteria  and  awarding 
authority. 

Applicable  Designator!  sf? 

Applicable  Gradefs)-’  If  grades  arc 
d liferent  for  various  designators,  break 
out  the  grades  for  each  designator. 


(Option  1 )  If  the  AQD  is  required  prior  to  Primary 

assignment  to  designated  billets,  then  use  the 

language:  “Billets  coded  with _ require  an 

officer  who  has  previously  been  awarded  the  Auxiliary-: 

_ code  prior  to  reporting  to  ultimate  duty 

assignment." 

(Option  2)  If  an  AQD  is  NOT  required  in  the 
performance  of  duties  for  designated  billets,  but 
is  to  be  assigned  only  after  successful 
completion  of  a  minimum  tour  length  in 
qualifying  billets,  then  the  AQD  will  be  used  for 
tracking  purposes  only.  In  this  case,  annotate 
this  block  “N/A.” 

Applicable  Designator! sV’ 


Length  of  validity'1  Does  the  AQD 
expire?  If  not,  use  “Indefinite."  If  so, 
identify-  timcframe/rcccrtification 
requirements. 


Applicable  Gradcfsl?  If  grades  are  different  for 
various  designators,  break  out  the  grades  for 
each  designator. 


Manpower  Type0  Is  this  AQD  for 
Active  Only,  Reserve  Only,  or  Active 
and  Reserve? 


Figure  10.  Template  for  AQD  Code  Table.  Source:  Department  of  the  Navy 

(2017a,  p.  D-5). 
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APPENDIX  C.  NAVY  TWI  ROI  MODEL 


Is  the  report  useful  for  the  audience? 


A 


Step  7:  Full 

' 

Were  all  costs  and  benefits  to  include 

Training  Report 

^ _ - 

intangible  benefits  considered? 

Step  6:  Measure  All 
Costs  and  Benefits 


k. 


How  was  the  knowledge  gained  used 
to  impact  the  Navy? 


Step  5:  Navy  Impact 

r - 

Was  knowledge  gained  and  retained  to 

use  in  their  utilization  tour? 

L 

S - -  J 

A 


Step  4:  TWI  Learning  and  Retention 


Were  participants  motivated  and  viewed 
TWI  as  relevant  and  value  added? 


Step  3:  Response  to  the  TWI 


How  doesNAVSUP  measure  performance 
and  can  we  use  it  to  determine  ROI? 


Step  2:  Link  TWI  ROI  to  Existing  NAVSUP 
Performance  Measurements 


Is  TOT  objectives  linked  to 
NAVSUP’s  objectives? 


Step  1 :  NAVSUP  Identify  Objectives  for  each  TWI  Location 


Figure  11.  Navy  TWI  ROI  Model.  Adapted  from  Schmidt  (2017). 
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APPENDIX  D.  DATA  COLLECTION  PLAN 


Data  Collection  Plan 

Program:  TWI  Responsibility:  Date: 

Level 

Objective  (s) 

Evaluation  Method 

Timing 

Responsibilties 

1 

Reaction,  Satisfaction  and  Planned 

Actions 

1)  Positive  Reaction 

2)  Recommended  Improvements 

3)  Action  Items 

1-3)  Initial  Program 
Evaluation  Form 

1-3)  Upon  completion  of  TWI  tour,  before  reporting  to  utilization 

tour 

1-3)  Sponsoring  company 
representative.  Email  a  copy  to 

PERS-4412. 

2 

Learning 

1)  Industry  innovative  business  approaches,  management 
practices,  and  corporate  organizational  culture 

2)  Industry  senior  and  executive  level  decision-making 

processes 

3)  Skills  in  private  sector  procedures  and  best  practices 

4)  Strategic  and  service-level  linkages  with  public  policy, 
corporate  and  non-governmental  organizations 

5)  Ability  to  embrace  changes  and  other  cultures  in  a  complex 
national  environment 

1-5)  Skill  Practice 

1-5)  During  TWI  tour 

1-5)  TWI  participants  and 
Sponsoring  company 
representative. 

3 

Job  Application 

1)  Meet  NAVSUP'sobjecives 

2)  Use  of  Skills 

3)  Frequency  of  Skill  Use 

4)  Barriers 

1)  Scorecard 

2-4)  Post  Program 
Evaluation  Form  (4-6 
months) 

1)  Upon  completion  of  TWI  tour,  before  reporting  to  utilization 

tour 

2-4)  4-6  months  after  working  at  utilization  tour 

1)  Sponsoring  company 
representative.  Email  a  copy  to 

PERS-4412. 

2-4)  PERS-4412 

4 

Business  Results 

1)  Cost  saving  to  the  Navy 

2)  Faster  and  more  efficient  ways  to  conduct  business 

3)  Increased  customer  satisfaction 

1-3)  Capture  Navy 

Impacts  Data 
(performance 
monitoring) 

1-3)  One  to  two  years  after  working  at  utilization  tour 

1-3)  PERS=4412  and  TWI 
patici  pants 
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APPENDIX  E.  TWI  SCORECARD 


TWI  SCORECARD 

NAVSUP  Strategic  Plan  2013-2017 

Objectives  for  [Home  Depot] 

YES 

NO 

Overall  Score 

Goal  1.  World  Class  Workplace  - 
Create  and  sustain  a  working 
environment  that  fosters 
teamwork  and  collaboration, 
rewards  innovation,  and  provides 
the  tools,  resources,  and 
developmental  opportunities 
employees  need  to  be  effective. 

Goal  2.  Unitv  of  Effort  -  Drive 
unity  of  effort  across  the  Naval 

Supply  Chain 

GREEN 

YELLOW 

G3:  Know  the  production  schedule 

Met  70%  or 

above  of  the 
objectives 

Met  40-69%  of 
the  objectives 

Met  39%  or  less 
of  the  objectives 

G3:  Understands  inventory  management 

G2:  Can  coordinate  with  third  party  transporters 

G3:  Can  manage  inventory  at  inventory  control  point 

G3:  Know  how  to  evaluate  work  processes  on  the  line 

G4:  Can  identify  technology  applications  and  purposes  to  improve  supply  chain  management 

Note:  Add  relevant  objective  and  tc 

ke  away 
calculate  the 

G4:  Know  the  benefits  and  problems  the  company  faces  with  new  software/management  programs 

irrelevance  objective  as  needed.  Tc 

G3:  Know  how  companies  utilize  strategic  sourcing 

relationships,  optimizing 
processes,  and  focusing  on 
customer  outcomes. 

Goal  3.  Effective,  Efficient 

G3:  Know  how  the  use  of  strategic  sourcing  affects  the  bottom  line 

Overall  Score  =  (#  yes /Total  #  of  objectives)  x  100 

G3:  Know  the  key  processes  in  the  operation 

Quality  Assurance 

G2  =  NAVSUP  Goal  2 

G3:  Can  test  for  quality  of  product 

G3  =  NAVSUP  Goal  3 

Performance  -  Refine  internal 

G3:  Know  the  inspection  techniques  used  by  the  company 

G4  =  NAVSUP  Goal  4 

business  processes  to  reduce 
operating  costs  and  ensure 
compliance  while  enhancing 
support  to  our  customers. 

Goal  4.  Data  Driven  Decision 

Making -Create  and  sustain  an 

information  environment  that 
leverages  technology  to  deliver 
greater  transparency,  facilitate 
information  data  sharing,  and 
enable  better  decision  making. 

Fuel 

G3:  Understands  the  environmental  factors  to  refinery  process 

G3:  Can  assess  fuel  testing  procedures 

G2:  Understands  the  operation  of  aviation  refueling  facilities 

G3:  Can  conduct  process  and  procedure  testing  to  ensure  safe  practices 

G3:  Can  coordinate  movement  of  fuel  assets 

Safety 

G3:  Improve  upon  emergency  response  plans 

Contract  Management 

G3:  Improve  organization's  structure  to  optimize  acquisition  function 

G2:  Know  the  types  of  contracts  used  in  procurement  and  its  effects  on  the  contract  choice 

G2:  Know  different  negotiation  strategies  used  by  industry 

G2:  Know  how  industry  carries  out  contract  administration 

G2:  Know  how  industry  manage  subcontractors 

G2:  Know  how  business  strategy  changes  when  dealing  with  the  government  versus  commercial 
customers 

G2:  Know  the  techniques  used  to  incentivize  contracts 

Process  improvement 

G3:  Know  how  Lean/  Six  Sigma  is  utilized  and  incorporated  into  business 

G3:  Know  different  improvement  initiatives  companies  are  implementing 

Gl:  Know  what  processes  companies  used  to  develop  leadership 

Gl:  Know  how  companies  instill  strategic  and  operating  excellence 

Adapted  from  NAVSUP  (n.d.,-b,  pp.  3-4). 
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APPENDIX  F.  INITIAL  PROGRAM  EVALUATION 


TWI  Program  Evaluation 

Please  help  us  evaluate  the  Training  with  Industry  (TWI)  program  by  answering  the  following  questions.  Give  the  completed  evaluation  to  your  industry 
representative  and  email  a  copy  to  NAVSUP  PERS-4412  Mr.  Phillip  Knauss  at  phillip.knauss@navy.mil.  Your  candid  feedback  will  be  key  to  improving  the 
program  for  future  participants. 

General 

1).  Give  examples  of  observed  innovative  business  approaches,  management  practices,  and  corporate  organizational  culture. 

Comments: 


2) .  Give  examples  of  observed  senior  and  executive  level  decision  making  processess  that  are  different  from  the  Navy  and/or  may  be  beneficial  for  the 
Navy. 

Comments: 

3) .  Give  examples  of  skills  developed  in  private  sector  precedures  and  best  practices  that  may  be  use  in  the  Navy. 

Comments: 


4).  Give  examples  of  how  you  contributed  to  the  strategic  and  service-level  linkage  with  public  policy,  corporate  and  non-government  organization. 
Comments: 


5).  Give  examples  of  how  your  ability  to  embrace  changes  and  other  cultures  in  a  complex  national  environment  has  enhanced. 
Comments: 


On  the  Job  Impact  and  Benefit 

6).  How  do  you  expect  to  use  what  you  learned  at  [FedEx]  in  the  Navy? 

Comments: 


7).  How  did  the  training  contribute  to  your  qualities  as  an  officer? 
Comments: 


8).  How  will  your  training  improve  your  quality  of  workforthe  Navy? 
Comments: 


9).  What  aspect  of  TWI  was  least  valuable  or  not  likely  to  be  used? 
Comments: 


Personal  and  Professional  Value 

10).  What  area  of  the  FITREP  performance  trait  is  most  impacted  by  the  TWI  Program? 
o  33.  Professional  Expertise:  Professional  knowledge  proficiency  and  qualification 

o  34.  Command  or  Organizational  Climate/Equal  Opportunity:  Contributing  to  growth  and  development,  human  worth,  community 
o  35.  Military  Bearing  Character:  Appearance,  conduct  physical  fitness,  adherence  to  Navy  Core  Values 
o  36.  Teamwork:  Contributions  toward  team  building  and  team  results 

o  37.  Mission  Accomplishment  and  Initiative:  Taking  initiative,  planning/prioritizing,  achieving  mission 
o  38.  Leadership:  Organizing,  motivating  and  developing  others  to  accomplish  goals. 

Comments: 


11).  How  did  this  training  contribute  to  your  professional  goals? 
Comments: 


TWI  Program  Improvement 

Please  rank  from  1  (Entirely  ineffective)  to  5(Very  effective) 

12).  Distribution  of  time  on  different  content  and  skill  areas  1  2  3  4  5 

Comments: 


13).  Length  of  the  program  and  the  balance  between  mentoring,  hands  on,  team  project,  etc  1  2  3  4  5 

Comments: 


14).  Design  and  facilitation  of  learning  environment  1  2  3  4  5 

Comments: 


15).  Connection  between  the  program  objectives  and  the  actual  learning  performance  1  2  3  4  5 

Comments: 


16).  What  should  be  added  to  the  program  to  enhance  the  learning?  Omitted? 
Comments: 
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APPENDIX  G.  POST-PROGRAM  EVALUATION 


TWI  Program  Evaluation  (4-6  Months  ) 

Please  help  us  evaluate  the  Training  with  Industry  (TWI)  program  by  answering  the  following  questions.  Email  to  NAVSUP  PERS-4412  Mr.  Phillip  Knauss  at 
phillip.knauss@navy.mil.  Your  candid  feedback  will  be  key  to  improving  the  program  for  future  participants  and  capturing  the  benefit  of  TWI. 

General 

1).  Give  examples  of  observed  innovative  business  approaches,  management  practices,  and  corporate  organizational  culture. 

Comments: 


2) .  Give  examples  of  observed  senior  and  executive  level  decision  making  processess  that  are  different  from  the  Navy  and/or  may  be  beneficial  for  the 
Navy. 

Comments: 

3) .  Give  examples  of  skills  developed  in  private  sector  precedures  and  best  practices  that  may  be  use  in  the  Navy. 

Comments: 


4).  Give  examples  of  how  you  contributed  to  the  strategic  and  service-level  linkage  with  public  policy,  corporate  and  non-government  organization. 
Comments: 


5).  Give  examples  of  how  your  ability  to  embrace  changes  and  other  cultures  in  a  complex  national  environment  has  enhanced. 
Comments: 


On  the  Job  Impact  and  Benefit 

6).  How  do  you  expect  to  use  what  you  learned  at  [FedEx]  in  the  Navy?  What  knowledge  and  skill  from  TWI  have  you  used? 
Comments: 


7).  How  did  the  training  contribute  to  your  qualities  as  an  officer? 
Comments: 


8).  Did  the  training  improve  your  quality  of  work  for  the  Navy?  How? 
Comments: 


9).  What  aspect  of  TWI  was  least  valuable  or  not  used? 
Comments: 


Personal  and  Professional  Value 

10).  Was  area/s  of  the  FITREP  performance  trait  impacted  by  the  TWI  Program? 
o33.  Professional  Expertise:  Professional  knowledge  proficiency  and  qualification 

o  34.  Command  or  Organizational  Climate/Equal  Opportunity:  Contributing  to  growth  and  development,  human  worth,  community 
o  35.  Military  Bearing  Character:  Appearance,  conduct  physical  fitness,  adherence  to  Navy  Core  Values 
o  36.  Teamwork:  Contributions  toward  team  building  and  team  results 

o  37.  Mission  Accomplishment  and  Initiative:  Taking  initiative,  planning/prioritizing,  achieving  mission 
o  38.  Leadership:  Organizing,  motivating  and  developing  others  to  accomplish  goals. 

Comments: 


11).  Did  the  training  contribute  to  your  expected  professional  goals? 
Comments: 


TWI  Program  Improvement 

Please  rank  from  1  (Entirely  ineffective)  to  5(Very  effective) 

12).  Distribution  of  time  on  different  content  and  skill  areas  1  2  3  4  5 

Comments: 


13).  Length  of  the  program  and  the  balance  between  mentoring,  hands  on,  team  project,  etc  1  2  3  4  5 

Comments: 


14).  Design  and  facilitation  of  learning  environment  1  2  3  4  5 

Comments: 


15).  Connection  between  the  program  objectives  and  the  actual  learning  performance  1  2  3  4  5 

Comments: 


16).  What  should  be  added  to  the  program  to  enhance  the  learning?  Omitted? 
Comments: 
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APPENDIX  H.  PROJECT  COST  SAVING 


Navy  Impacts 

Cost  Saving  (dollar,  time,  etc) 

Cost  Saving  in  dollars 

Confidence  Level 

Benefit  Value 

i 

Example:  improved  warehouse 
restocking  time  by  50% 

Example:  Decrease  the 
time  by  50%  saved 
$50,000 

Example:  Used  the  method  learned  from  Home  Depot  while 
conducting  TWI  so  is  95%  confindence  this  saving  is  due  to 
knowledge  learned  at  TWI 

Example:  At  95%  confidence 
indicate  at  error  of  5%  so  the 
range  of  benefit  is  from  $47,500 
to  $52,500.  $47,500  is  use  for  a 
more  conservative  estimate. 

2 

3 

Total  Benefits  to  be  added  to  Step  6  CBA 
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APPENDIX  I.  FULL  REPORT 


Training  with  Industry 

Return  on  Investment 

ROI  88% 

For  each  dollar  invested  in  the  TWI  program,  the  Navy  received  $0.88  dollars  in  return  after  the  cost  of  the  program  had  been 
recovered 

Benefits  and  Costs  Breakdown  Data  From  Follow  Action 

POC 

Due  By 

Benefits 

1)  Productivity  at  5%; 
Capability  Gaps  at 
20% 

$619,041.00 

2)  Project  cost  saving 

$0.00 

Use  Navy  Impacts  form  to  capture  data  from  TWI  participant  in  the  future 

PERS-4412 

Costs 

1)  One  year  tour 

$707,036.00 

FY2017  DOD  Military  Personnel 

2)  Selection  Board 

$16,860.08 

Cannot  be 

Intangible  Banefits  YES  NO  Data  From  Follow  Action 

Determined 

POC 

Due  By 

1)  Fulfilling  identified  Capability  Gaps 

X 

Update  MOU/MOA  and  use  initial  Program  Evaluation  Form  for  future 

Survey 

participants 

PERS-4412 

2)  Meeting  NAVSUP  Objectives 

X 

Update  MOU/MOA  and  use  initial  Program  Evaluation  Form  and  Scorecard 
Survey  ,  , 

Form  forfuture  participants 

PERS-4412 

3)  Professionally  a  better  officer  afterTWI  tour 

X 

No  further  action  require,  selection  board  is  doing  a  sufficient  job  of 

Survey 

selecting  participant  interested  in  making  the  Navy  a  career 

4)  Improved  civil-military  relation 

- 

5)  Increase  in  job  satisfaction 

X 

No  further  action  require,  selection  board  is  doing  a  sufficient  job  of 

Survey 

selecting  participant  interested  in  making  the  Navy  a  career 

6)  Recruiting 

X 

Areas  for  Improvements  Data  From  Follow  Action 

POC 

Due  By 

1)  Assign  real  world  project  to  complete  to  maximize  business 
impact;  better  utilization  of  learned  skills 

Survey  Change  selection  process  and  TWI/Utilization  tour  format 

PERS-4412 

2)  Uniform  strategic  leadership  level  interaction  across  all 
partnership  companies 

Survey  Update  MOU/MOA  for  clearer  objectives  and  responsibility  of  each  party 

PERS-4412 

3)utilize  forms  provided  to  better  measure  costs  and  benefits 

MBA  Project  Update  MOU/MOA  for  clearer  objectives  and  responsibility  of  each  party 

PERS-4412 
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APPENDIX  J.  SURVEY  TO  TWI  PARTICIPANTS 


1.  How  many  years  had  you  been  in  the  Navy  before  your  PCS  to  your  TWI  location? 
Ol-5  years 

06  -8  years 
08-10  years 
OlO  -12  years 
_  More  than  12  years 

2.  What  was  your  paygrade  when  you  PCS'd  to  your  TWI  location? 

002 

003 

004 

005 

3.  Did  you  have  dependents  at  the  time  of  your  PCS  to  your  TWI  location? 

OYes 

ONo  (Skip  to  question  6) 

4.  Did  dependents  PCS  with  you  to  the  TWI  location? 

OYes 

ONo  (Skip  to  question  6) 

5.  On  a  scale  of  1  to  10,  did  the  two  PCS  moves  in  a  one  year  period  (to  and  from  TWI) 
impact  family? 

1  -  Very  Negative  Experience;  10  -  Very  Positive  Experience 

123456789  10 

Impact  on  family  _ 1 

6.  With  what  company  did  you  complete  your  TWI? 

OThe  Home  Depot 

OFedEx 

OStarbucks 

OExxonMobil 

7.  How  many  months  was  your  TWI  internship? 

Oil  months 

Ol2  months 
OOther 

8.  Did  you  have  a  graduate  degree  prior  to  TWI? 

OYes 

ONo  (Skip  to  question  11) 
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9.  Did  you  owe  the  Navy  a  payback  tour  for  the  graduate  degree? 

OYes 

QNo 

10.  What  subspecialty  did  you  obtain  with  your  graduate  degree? 

( _  Contracting 

OFinancial  Management 
OSupply  Chain  Management 
OOther 

11.  Selection  to  TWI  was  through: 

OSelf  Nomination 

OSelected  without  self  nomination 

12.  Did  selection  to  TWI  have  any  impact  on  your  career  plans? 

ONo,  my  plan  has  been  to  serve  at  least  20  years 

OYes,  I  was  planning  on  getting  out  of  the  Navy  prior  to  selection.  But  now  I  plan  to  stay 
through  my  payback  tour  and  at  least  another  tour. 

.  1  am  planning  to  get  out  of  the  Navy  after  my  payback  tour. 

I  :I  am  requesting  a  waiver  of  a  payback  tour  and  want  to  get  out  of  the  Navy  now. 
OOther 

13.  Have  you  completed  the  required  utilization  tour  after  TWI? 

OYes 

!_  Currently  in  my  utilization  tour 
ONo  (Skip  to  question  17) 

14.  At  what  command  did  you  complete  your  utilization  tour? 

15.  Did  you  know  prior  to  starting  TWI  where  your  utilization  tour  would  be? 

OYes 

ONo 

16.  Did  you  have  a  choice  in  where  you  would  complete  your  utilization  tour? 

OYes 

ONo 

17.  On  a  scale  of  1  to  10,  how  did  you  feel  about  selection  to  TWI  prior  to  attending  the 
internship? 

How  did  you  feel  about  TWI  after  completing  the  internship? 

1  -  Very  Upset;  10  -  Very  Excited 

123456789  10 

Prior  to  attending  TWI 
After  completion  of  TWI 
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18.  Please  explain  any  change  in  feelings  regarding  TWI  from  question  17. 

19.  Do  you  think  the  value  added  from  TWI  is  evident  in  the  immediate  follow-on  tour  or 
are  (will)  the  benefits  be  more  evident  in  more  senior  billets,  two  or  three  tours  after 
TWI? 

Ol  see  (saw)  the  value  immediately 

Ol  think  I  will  see  (saw)  more  value  in  more  senior  billets 

20.  How  would  you  rate  the  interaction  with  TWI  executives  as  value  added  to  your 
military  career? 

Please  explain  why  in  the  comment  box. 

QTo  a  very  great  extent 
_  To  a  great  extent 
C  To  a  small  extent 
C  To  no  extent 

21.  What  did  you  find  most  valuable  from  your  TWI  experience? 

22.  What  experiences  gained  through  TWI  do  you  feel  could  not  have  been  gained  in  a 
traditional  Supply  Corps  billet? 

Please  choose  all  that  apply: 

□Strategic  thinking  skills 
□Critical  thinking  skills 
□Analysis  skills 
□Applying  technology 
□Decision  making 
□Executive  decision  making 
□Supply  chain  operations 
□Contingency  planning 
□Other 

23.  Were  you  able  to  identify  best  business  practices  from  TWI  that  would  be  applicable 
to  the  Navy? 

OYes 

ONo  (Skip  to  question  25) 

24.  Did  you  have  the  opportunity  to  implement  into  the  Navy  the  best  business  practices 
identified  from  TWI? 

OYes 

ONo 

25.  Do  you  feel  you  lost  any  edge  in  military  knowledge  during  your  time  away  from  the 
Fleet? 

OYes 

ONo 
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26.  The  student  researchers  request  your  consent  to  obtain  the  following  information 
from  your  personnel  record  at  Naval  Personnel  Command: 

Year  selected  for  TWI 

Rank  and  years  of  service  when  selected 

Promotion  history  after  TWI 

Number  of  years  in  service  to  include  the  present 

If  separated  or  retired,  how  long  after  TWI  did  participant  leave  the  service 

All  names  will  be  removed  prior  to  data  analysis.  Data  will  be  used  to  determine  any 
effects  TWI  has  on  promotion  or  retention. 

If  you  give  your  consent,  please  type  your  name  in  the  box  for  student  researchers  to 
provide  to  PERS  for  data  retrieval.  If  you  do  not  give  consent,  type  No. 
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APPENDIX  K.  SURVEY  SUMMARY 


A  26  question  survey  was  sent  to  past  TWI  participants  to  gather  reactions  to 
training,  utilization  of  training,  and  impact  to  career  milestones.  The  survey  was  able  to 
solicit  15  of  the  24  TWI  participants  (62.5%). 

A  number  of  questions  were  of  a  general  manner  and  provided  support  for  some 
rec  ommendations . 

Questions  2,  7,  8,  and  13  -  16  were  to  determine  if  the  Supply  Corps  is  meeting  its 
own  selection  criteria  and  the  DOD  criteria  laid  out  in  DOD  Instruction  1322.06. 
Education,  rank,  length  of  tour,  and  assignment  to  a  utilization  tour  are  all  being  met. 

Questions  17  -  21  were  to  seek  the  participants’  response  to  the  TWI  program. 
Eleven  of  the  15  responses  (73%)  saw  value  in  the  program  immediately  with  their 
follow-on  tour,  while  four  expressed  a  greater  value  from  TWI  in  more  senior  billets. 

Questions  9,  12,  19,  and  22  -  23  were  to  determine  learning  and  retention  from 
TWI.  Thirteen  of  the  15  respondents  (87%)  were  able  to  identify  best  business  practices 
from  their  TWI  location  to  bring  back  to  the  fleet. 

Question  24  related  to  TWI’s  impact  to  the  Navy.  Of  the  13  participants  who 
answered  positively  to  identifying  best  business  practices  for  use  to  the  Navy,  a  total  of 
ten  responded  to  the  follow-on  question  about  having  the  opportunity  to  implement  better 
business  practices  into  the  Navy.  Seven  of  the  ten  (70%)  answered  yes  they  had  the 
opportunity  to  implement  the  best  business  practices  into  the  Navy. 
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